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Modeling  and  Analysis  of  Networked  Control  Systems 
using  Stochastic  Hybrid  Systems* 

Joao  P.  Hespanha  ^ 

September  3,  2014 


Abstract 

This  paper  aims  at  familiarizing  the  reader  with  Stochastic  Hybrid  Systems  (SHSs)  and 
enabling  her  to  use  these  systems  to  model  and  analyze  Networked  Control  Systems  (NCSs). 
Towards  this  goal,  we  introduce  two  different  models  of  SHSs  and  a  set  of  theoretical  tools  for 
their  analysis.  In  parallel  with  the  presentation  of  the  mathematical  models  and  results,  we 
provide  a  few  simple  examples  that  illustrate  the  use  of  SHSs  to  models  NCSs. 

Keywords:  Network  Control  Systems;  Hybrid  Systems;  Stochastic  Processes;  Stability;  Markov 
Processes 

1  Networked  Control  Systems 

The  expression  Networked  Control  Systems  (NCSs)  typically  refers  to  feedback  control  systems  for 
which  some  of  the  sensors,  controllers,  and  actuators  communicate  with  each  other  using  a  shared 
communication  network.  The  use  of  a  multi-purpose  shared  network  reduces  installation  and  main¬ 
tenance  costs  and  adds  flexibility,  as  it  permits  the  system  reconfiguration  and/or  expansion  with 
minimal  additional  infrastructure  costs.  In  view  of  this,  NCSs  are  finding  application  in  numerous 
areas  that  include  the  automotive  industry,  the  aviation  industry,  robotics,  process  control,  and 
building  control,  among  others. 

While  NCSs  are  attractive  from  the  perspective  of  cost  of  deployment  and  maintenance,  they 
introduce  significant  design  challenges,  because  the  traditional  unity  feedback  loop  that  operates  in 
continuous  time  or  at  a  fixed  sampling  rate  is  not  adequate  when  sensor  data  arrives  from  multiple 
sources,  asynchronously,  delayed,  and  possibly  corrupted.  Consequently,  NCSs  have  been  the  focus 
of  intense  study  in  the  last  few  years  [16,  25,  29]. 

This  paper  is  focused  on  two  aspects  of  NCSs  that  are  responsible  for  important  challenges 
in  analyzing  and  designing  NCSs  and  that  are  prompting  the  development  of  new  formal  tools  to 
study  these  systems. 

*This  paper  is  partially  based  on  material  presented  in  a  plenary  lecture  at  the  4th  IFAC  Workshop  on  Distributed 
Estimation  and  Control  in  Networked  Systems  (NECSYST3). 

Mhis  material  is  based  upon  work  supported  by  the  National  Science  Eoundation  under  Grants  No.  EE- 11 37835 
and  CNS-1329650,  and  by  the  Institute  for  Collaborative  Biotechnologies  through  grant  W911NE-09-0001  from  the 
U.S.  Army  Research  Office. 
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controller 


Figure  1:  Networked  Control  System  (NCS) 


1.  In  NCSs,  the  information  that  flows  between  different  elements  in  the  feedback  loop  (sensors, 
actuators,  or  controllers)  is  carried  in  atomic  units  called  packets]  and  the  departure  and 
arrival  of  these  packets  causes  instantaneous  changes  in  the  state  of  the  system.  Moreover, 
the  protocols  used  to  process  packets  often  contains  discrete  or  logic  variables,  which  must 
be  taken  into  account  in  their  analysis. 

2.  The  shared  network  that  supports  the  communication  between  the  different  elements  of  a  NCS 
introduces  uncertainty  that  typically  arises  from  other  (unmodeled)  systems  that  compete  for 
network  resources  as  well  as  from  stochasticity  introduced  explicitly  by  the  network  protocols 
(e.g.,  random  backoffs  in  response  to  packet  collisions). 

The  integration  of  continuous  dynamics  with  discrete  variables  (often  boolean)  requires  the  use 
of  “hybrid”  models  for  dynamical  systems,  whose  solutions  should  include  “stochastic  behavior” 
to  capture  the  uncertainty  common  in  NCSs.  These  requirements  motivate  the  use  of  Stochastic 
Hybrid  Systems  (SHSs)  as  a  natural  modeling  formalism  for  NCSs. 

The  goal  of  this  tutorial  paper  is  twofold:  First,  we  aim  at  familiarizing  the  reader  with  SHSs  as 
a  modeling  framework  for  NCSs.  Second,  we  want  to  introduce  a  few  theoretical  tools  that  we  found 
instrumental  for  the  analysis  and  design  of  NCSs  feedback  controllers  and  network  protocols.  We 
shall  attempt  to  make  the  presentation  tutorial  but  rigorous,  and  incorporate  in  it  a  few  examples 
to  illustrate  the  use  of  the  models  and  tools. 

Towards  the  above  mentioned  goals.  Section  2  introduces  the  concept  of  SHSs.  We  start  with 
a  deterministic  hybrid  model,  whose  state  has  a  component  that  evolves  continuously  and  another 
that  is  discrete  in  nature,  changing  through  instantaneous  resets  or  impulses  (Section  2.1).  The  key 
mechanism  used  here  to  introduce  stochasticity  is  through  the  times  at  which  impulses  take  places, 
which  in  the  context  of  networked  control  systems,  often  correspond  to  the  times  at  which  messages 
are  sent  or  received  from  the  network.  In  the  first  (and  simpler)  model  considered,  denoted  by  Time- 
triggered  Stochastic  Hybrid  Systems  (TTSHSs)^  the  reset  times  may  causally  depend  on  the  hybrid 
system’s  discrete  state,  but  are  not  allowed  to  depend  on  its  continuous  state  (Section  2.2).  This 
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restriction  is  removed  in  a  second  model  for  stochastic  hybrid  systems,  denoted  by  State-Driven 
Stoehastie  Hybrid  Systems  (SDSHSs)^  where  a  (causal)  dependence  on  the  full  state  is  allowed 
(Section  2.3).  For  both  models,  we  provide  algorithms  to  construct  the  sample  paths  for  the  SHSs. 
These  algorithms  provide  the  basis  for  the  formal  analysis  of  these  systems  and  can  also  be  used 
to  generate  Monte  Carlo  simulations  for  the  SHSs. 

In  parallel  with  the  introduction  of  formal  models  for  SHSs,  we  provide  two  examples  that  illus¬ 
trate  the  use  of  SHSs  to  model  networked  control  systems.  One  example  consists  of  a  feedback  loop 
with  multiple  sensors  that  communicate  with  a  controller  through  a  shared  network  (Example  1) 
and  the  other  refers  to  the  estimation  of  the  state  of  a  system  based  on  measurements  received 
through  a  packet-switched  network  (Example  2). 

The  formal  tools  presented  here  to  analyze  SHSs  are  focused  on  establishing  global  existence  of 
solution,  stability,  and  on  constructing  bounds  on  the  state  of  the  system.  There  is  a  wide  range 
of  stability  notions  applicable  to  SHSs  [23]  and  we  review  some  of  the  most  commonly  used  ones 
in  Section  3.  The  stability  notions  considered  can  be  classified  in  two  broad  categories:  bounds  on 
the  probability  that  the  state  of  the  system  “misbehaves”  or  bounds  on  the  expected  value  of  the 
state. 

The  stability  theory  for  linear  TTSHSs  is  fairly  well  established  and  simple  necessary  and 
sufficient  stability  conditions  are  available.  These  conditions  are  presented  in  Section  4  for  the  case 
of  a  single  discrete  state,  but  they  have  been  generalized  to  multiple  discrete  states  [3]. 

The  key  technical  tools  discussed  in  Section  5  for  the  formal  analysis  of  nonlinear  TTSHS  and 
SDSHS  are  based  on  the  concepts  of  extended  generator  and  Lyapunov  function.  Essentially,  the 
extended  generator  of  a  SHS  provides  a  formula  (known  as  Dynkin’s  formula)  that  can  be  used 
to  compute  the  instantaneous  value  of  the  derivative  of  the  expected  value  of  a  function  V  that 
dependes  on  the  (continuous  and  discrete)  state  of  the  SHS;  much  like  the  chain  rule  allows  one  to 
compute  the  time  derivative  of  a  function  V {x)  along  solutions  to  a  (deterministic)  ODE  of  the  form 
X  =  f{x)  (Section  5.1).  Section  5.3  provides  several  Lyapunov-based  conditions  that  can  be  used 
to  establish  global  existence,  stability,  and  boundedness  for  SHSs.  These  conditions  are  expressed 
in  terms  of  algebraic  inequalities  on  “candidate  Lyapunov  functions”  that  greatly  resemble  the 
conditions  that  appear  in  Lyapunov’s  stability  theorem  for  deterministic  ODEs. 

While  the  extended  generator  and  Dynkin’s  formula  are  powerful  tools  for  the  analysis  of  SHSs, 
their  direct  applicability  requires  stringent  assumptions,  which  often  cannot  be  established  indepen¬ 
dently.  A  solution  to  this  technical  difficult  relies  in  considering  a  “stopped”  process  that  matches 
the  solution  to  the  SHS  until  the  first  time  it  leaves  a  bounded  set  and  gets  “frozen”  at  the  precise 
time  it  leaves  the  set.  Two  key  features  make  stopped  processes  especially  useful:  (i)  Dynkin’s 
formula  holds  for  stopped  processes  under  very  mild  assumptions  and  (ii)  the  stopped  process  con¬ 
verges  to  the  state  of  the  hybrid  system  as  one  enlarges  the  set  outside  which  the  process  gets 
frozen.  The  use  of  stopped  processes  to  study  SHSs  is  discussed  in  Section  5.2.  While  the  main 
goal  of  this  paper  is  not  to  provide  the  full  formal  proofs  behind  the  tools  prevented,  we  include  in 
Section  5.4  the  key  steps  behind  the  Lyapunov-based  proofs.  Our  goal  is  to  provide  the  reader  with 
the  basic  tools  needed  to  construct  variations  of  the  results  presented  here,  instead  of  attempting  to 
provide  a  long  (and  somewhat  boring)  list  of  all  possible  variations.  However,  the  stability  results 
presented  in  Section  5.3  are  self-contained  in  that  using  these  results  does  not  require  the  discussion 
in  Sections  5.2  and  5.4. 
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The  present  paper  aims  at  being  a  tutorial  on  SHSs  and  not  an  exhaustive  literature  survey  on 
SHSs.  For  the  latter,  we  refer  the  reader  to  the  excellent  recent  survey  [23]. 

Notation  All  random  variables  in  this  write-up  are  measurable  on  the  same  probability  space 
P).  We  denote  random  variables  and  stochastic  processes  in  boldface  and,  for  short  of 
notation,  we  generally  omit  the  dependence  on  the  outcome  cj  e  12  [as  in  tk  or  N(t),  instead 
of  or  N(t;6(;)].  In  expressions  involving  random  variable  that  require  some  quantification  on 

wpo 

cj,  we  use  the  superscript  •  to  denote  universal  quantification  with  respect  to  some  subset  of  12 
with  probability  one  [as  in  X  Z  to  express  that  312  e  T  such  that  P(12)  =  1  and  X((x;)  =  Y (cj), 
Vcj  G  12.]  A  time-dependent  function  x  :  [0,T)  ^  M^,  T  e  (0,oo]  that  is  continuous  from  the  right 
and  has  well-defined  limits  from  the  left  for  every  time  t  e  [0,  T)  is  said  to  be  cadlag  (from  French 
“continue  a  droite,  limite  a  gauche”).  For  cadlag  functions,  we  denote  by  (•)“  the  left-limit  operator 
that  maps  x  into  the  function  x~  :  (0,T)  ^  defined  by  x~{t)  •=  lim^^tx{r)^  Mt  e  (0,T). 

2  Stochastic  Hybrid  Systems 

Hybrid  Systems  (HSs)  combine  continuous-time  dynamics  with  discrete  modes  of  operation.  The 
states  of  such  systems  usually  have  two  distinct  components:  one  that  evolves  continuously,  typ¬ 
ically  according  to  a  differential  equation;  and  another  one  that  is  discrete  and  changes  through 
instantaneous  resets  or  impulses.  The  Stochastic  Hybrid  Systems  (SHSs)  considered  here  can  be 
viewed  as  HSs  for  which  the  resets  are  triggered  by  stochastic  events,  much  like  transitions  between 
states  of  a  continuous-time  Markov  chains.  However,  the  rate  at  which  resets  occur  is  allowed  to 
depend  on  both  the  continuous  and  the  discrete  states  of  the  SHS. 

SHSs  have  been  introduced  in  [17]  and  further  developed  in  [6,  7,  14,  15,  22].  Here,  we  consider 
a  restricted  form  of  SHSs  that  are  closely  related  to  (and  heavily  inspired  by)  the  Piecewise- 
Deterministic  Markov  Process  (PDMPs)  introduced  by  Davis  [11]  and,  in  fact,  our  SHSs  can  be 
viewed  as  a  special  case  of  PDMPs  and  thus  inherent  many  of  the  PDMPs  properties.  SHSs  are 
also  closely  related  to  Markov  Jump  Linear  Systems  (MJLS)  [10,  20]  and  to  Switching  Difusions 
(SD)  [13,  27],  which  differ  from  our  SHSs  in  that  the  emphasis  in  MJLSs  and  SDs  is  in  the  change 
in  dynamics  at  a  set  of  event  times  and  not  on  the  impulsive  effects  that  are  fundamental  in 
many  NCSs.  Nevertheless,  MJLSs  have  been  successfully  used  to  study  fairly  complex  NCSs  [8]. 
Also,  in  SDs,  the  solution  between  discrete  events  is  obtained  from  a  stochastic  differential  equation, 
instead  of  an  ordinary  differential  equation;  therefore  such  systems  exhibit  randomness  even  without 
resets.  The  reader  is  referred  [23]  for  a  recent  comprehensive  survey  on  SHSs  that  discusses  the  key 
differences  between  these  and  related  models  that  have  appeared  in  the  literature. 

In  the  remainder  of  this  section,  we  first  introduce  a  simple  model  for  deterministic  hybrid 
systems  and  then  explore  a  few  mechanisms  to  introduce  stochasticity  into  their  solutions  that  are 
particularly  useful  in  the  context  of  NCSs. 

2.1  Deterministic  Hybrid  Systems 

We  consider  hybrid  systems  (HS)  whose  continuous  state  x{t),  t  ^  0  takes  values  in  and  whose 
discrete  state  q{t),  t  ^  0  takes  value  in  a  (typically)  discrete  state  Q.  When  q{t)  takes  some  value 
qi  ^  Q,  we  say  that  the  HS  is  in  mode  qi  and  when  q{t)  changes  from  some  qi  e  Q  to  another 
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q2  ^  Q,  we  say  that  the  HS  transitioned  from  mode  qi  to  mode  ^2-  A  HS  for  which  Q  has  a  single 
element  (and  therefore  the  system  is  always  in  the  same  mode)  is  called  an  impulsive  system  (IS). 
In  these  hybrid  systems,  q{t)  is  necessarily  always  equal  to  the  only  element  of  Q  and  therefore  we 
generally  omit  any  reference  to  their  discrete  state. 

The  simple  model  of  deterministie  hybrid  systems  (DHSs)  considered  here  requires  three  ingre¬ 
dients: 

1.  a  family  of  (loeally  Lipsehitz)  veet or  fields  {fq  :  ^  W^^q  e  Q}  that  defines  the  evolution 

of  the  continuous  state  x{t)  in  each  mode  q  e  Q] 

2.  a  family  of  reset  maps  {(/)£  :  Q  x  Q  x  e  C}  that  characterizes  how  the  resets  or 

impulses  change  the  system’s  state;  and 

3.  a  family  of  guards  {gi  :  Q  x  £  e  C}  that  defines  when  resets  or  impulses  take  place. 

The  families  of  reset  maps  and  guards  are  parameterized  by  the  same  (typically  finite)  set  C  so 
that,  for  each  i  e  the  guard  gi  is  associated  with  the  reset  map  0^. 


{q,x)  ^ 


(a)  (Multi-mode)  Hybrid  System  (HS) 


gi{x)  ^  0 


92{x)  ^  0 


(b)  Impulsive  System  (IS) 


Figure  2:  Deterministic  Hybrid  Systems  (DHS) 

Figure  2  shows  a  graphical  representation  of  a  DHS.  Each  elipse  corresponds  to  a  specific  value 
of  the  discrete  state  in  Q  and  is  labelled  with  the  differential  equation  for  the  continuous  state  x(t) 
within  the  corresponding  mode.  The  arrows  are  associated  with  transitions  between  modes  and 
each  arrow  corresponds  to  an  element  of  C  (and  therefore  to  a  pair  guard/reset  map).  The  guard 
condition  ge{q^x)  ^  0  appearing  at  the  start  of  an  arrow  indicates  when  the  transition  is  triggered 
and  the  reset  map  (q^x)  ^  (j)i{q.^x)  appearing  at  the  end  of  the  arrow  specifies  how  the  transition 
changes  the  system’s  state. 

To  construct  a  solution  to  this  DHS,  starting  at  an  initial  condition  g(0)  =  qo  e  x(0)  = 
xq  e  M^,  we  solve  the  differential  equation  corresponding  to  the  mode  qo  until  one  of  the  guard 
conditions  becomes  active  (i.e.,  until  one  of  the  gi  becomes  positive  or  zero).  At  that  time,  we 
reset  the  state  according  to  the  corresponding  reset  map  (fe  and  proceed  by  solving  the  differential 
equation  corresponding  to  the  new  mode.  This  procedure  can  be  formalized  as  follows: 

1.  Set  A:  =  0,  to  =  0. 
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2.  Solve  the  initial-value  problem  z{tk)  =  x{tk)^  z  =  ^  ^.nd  let  [tk^T)  denote  its 

maximum  interval  of  existence. 

3.  If  any  one  of  the  gi{q{tk)^  z{t))  is  non- negative  at  time  or  two  or  more  of  the  gi{q{tk)^  z{t)) 

became  non- negative  simultaneously  in  the  interval  then  this  procedure  fails 

4.  If  a  single  gi{q{tk)^  z{t))  ^  le  C  becomes  non-negative  in  the  interval  i.e., 

4+1  :=  inf  {t  e  (4,T)  :  e  £,  ge{q{tk),  z{t))  ^  O} 
belongs  to  the  (open)  interval  and  e  C  such  that 

9eu{q{tk),z{tk+i))  >  0,  ge{q{tk),  z{tk+i))  <0,  ^  4; 

then  define  the  DHS  state  in  the  interval  [tk^tk+i)  to  be 

q{t)  =  q{tk),  x{t)  =  z(t),  Vt  G  [4,4+i) 
and  at  the  time  to  be 

(g(4+i),a:(4+i))  =  04(g(4),2:(4+i))  (  =  </)4  (g"(4+i),  a;“(4+i))  )  (1) 

Increment  the  integer  k  and  go  back  to  2. 

5.  If  none  of  the  gi{q{tk)^  z{t))  ^  £  e  C  becomes  non-negative  in  the  interval  [tk^T)^  then  define 
the  DHS  state  in  the  interval  [tk^T)  to  be 

q{t)  =  qitk),  x{t)  =  z{t),  vt  e  [4,  T) 


and  terminate. 

The  times  tk  are  called  reset  times  and  this  solution  is  defined  up  to  a  maximal  time  T^ax  that 
can  be  either  equal  to  T  if  the  construction  terminate  through  5  or  sup^  tk  if  it  does  not  terminate. 
In  this  and  in  all  subsequent  constructions  of  solutions  to  HSs  the  state  variables  x{t)  and  q{t)  are 
typically  not  continuous  at  the  reset  times  tk^  but  they  are  continuous  from  the  right  and  have 
well-defined  limits  from  the  left.  Signals  of  this  type  are  called  eadlag  (from  French  “continue  a 
droite,  limite  a  gauche”). 

Many  models  for  DHSs  have  been  proposed  in  the  literature.  The  one  described  here  is  a 
simplification  of  previous  models  and,  most  notably,  lacks  non-determinism,  i.e.,  the  ability  to 
allow  solutions  that  may  or  may  not  reset  when  guards  become  active.  The  construction  outlined 
above  leads  to  uniqueness  of  solution,  which  somewhat  simplifies  the  introduction  of  stochasticity. 
However,  non-determinism  can  be  combined  with  stochasticity  as  shown  in  [22]. 

^One  can  formulate  conditions  that  are  sufficient  for  the  construction  not  to  fail,  but  since  the  emphasis  of  this 
paper  is  in  stochastic  hybrid  systems,  we  do  not  pursue  that  here. 
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2.2  Time- Triggered  Stochastic  Hybrid  Systems 

In  the  DHS’s  considered  in  Section  2.1,  the  times  tk  at  which  the  state  is  reset  according  to  (1) 
correspond  to  the  times  at  which  one  of  the  guards  gi  becomes  non- negative.  In  Time- Triggered 
Stoehastie  Hybrid  Systems  (TTSHS)^  these  t^s  are  random  variables  with  the  property  that  the 
intervals  '=  —  tk^  e  between  consecutive  resets  are  independent  random  variables 

with  a  probability  distribution  that  may  depend  on  the  current  mode.  These  hybrid  systems  are 
also  known  as  Stochastic  Hybrid  Systems  with  Renewal  Transitions  [2,  3]  because  the  stochastic 
process  N(T)  that  counts  the  number  of  resets  tk  in  the  interval  (0,T]  is  a  renewal  process. 

The  model  of  a  TTSHS  thus  requires  three  ingredients: 

1.  a  family  of  (loeally  Lipsehitz)  veet or  fields  {fq  :  ^  e  Q}  that  defines  the  evolution 

of  the  continuous  state  x{t)  in  each  mode  q  ^  Q] 

2.  a  family  of  reset  maps  {(/)£  :  Q  x  Q  x  e  C}  that  characterizes  how  the  resets  or 

impulses  change  the  system’s  state;  and 

3.  a  family  of  reset-time  distributions  {/a^  :  £  e  Q  ^  Q}  that  defines  when  resets  or  impulses 
take  place. 

Also  here,  the  families  of  reset  maps  and  reset-time  distributions  are  parameterized  by  the  same 
(typically  finite)  set  C  so  that,  for  each  £  e  the  reset-time  distributions  piq^  q  ^  Q  are  associated 
with  the  reset  map  0^. 


Figure  3:  Time- Triggered  Stochastic  Hybrid  System  (TTSHS) 

Figure  3  shows  a  graphical  representation  of  a  TTSHS:  The  key  difference  with  respect  to  the 
DHS  in  Figure  2  is  that  now  the  guard  conditions  have  been  replaced  by  the  random  variables  tk 
that  represent  the  reset  times,  which  are  determined  by  the  reset-time  distributions  f/q^  £  e  q  ^  Q- 

To  construct  a  solution  to  this  TTSHS,  starting  at  an  initial  condition  q(0)  =  q^)  e  x(0)  = 

xo  e  we  draw  one  random  number  hg,  £  g  £  for  each  reset-time  distribution  £  ^  C 

and  solve  the  differential  equation  corresponding  to  the  mode  go  foi*  a  time  interval  equal  to  the 
smallest  of  the  hg.  When  that  interval  elapses,  we  reset  the  state  according  to  the  reset  map  (fi 
corresponding  to  the  smallest  hg  and  proceed  by  solving  the  differential  equation  corresponding 
to  the  new  mode.  Formally,  the  construction  of  the  solution  to  a  TTSHS  is  analogous  to  the 
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construction  described  in  Section  2.1  for  a  DHS,  but  with  the  new  (stochastic)  mechanism  used  to 
generate  the  reset  times.  Specifically,  the  steps  2-5  should  now  be  replaced  by: 

2’.  Solve  the  initial-value  problem  z  =  /q(^^)(z),  Vt  ^  tk]  and  let  denote 

its  maximum  interval  of  existence. 

3’.  Draw  independent  random  numbers  h^,  £  e  C  with  distributions  If  of  fho  is 

equal  to  zero  or  more  than  one  of  the  is  equal  to  hk  •=  min^g^  then  this  procedure  fails. 
Otherwise  set 


tfc+i  +  hfc- 


4’.  If  tk^i  ^  (tfc,  T)  and  a  single  is  equal  to  hk^  i.e.,  3£k  ^  ^  such  that 


hj*  =  hi, 


hi  >  hfc,  V£  #  £k] 


then  define  the  TTSHS  state  in  the  interval  [tfc,tfc+i)  to  be 

q(t)  =  q(tfc),  x(t)  =  z(t),  Vt  e  [t^,  t^+i) 
and  at  the  time  t^+i  to  be 

(q(tfc+i),x(tfc+i))  =  04(q(tfc),z(tfc+i))  (  =  </)4  (q“(tfc+i),  x“(tfc+i)) 

Increment  the  integer  k  and  go  back  to  2’. 

5’.  If  tk+i  ^  T,  then  define  the  TTSHS  state  in  the  interval  [tk^  T)  to  be 

q(t)  =  q(tfc),  x(t)  =  z(t),  Vt  e  [tfc,T) 


(2) 

(3) 

(4) 


and  terminate. 

Also  here,  the  random  times  tk  are  called  reset  times  and  this  solution  is  defined  up  to  a  maximal 
time  Tjnax  that  can  be  either  equal  to  T  if  the  construction  terminates  through  5’  or  snpj^tk  if  it 
does  not  terminate. 

The  following  assumptions  can  be  used  to  make  sure  that  this  construction  does  not  fail  in  step 
3’  with  probability  I;  either  because  we  get  an  equal  to  zero  or  because  more  than  one  of  the 
is  equal  to  hk  •=  min^e^  h^. 

Assumption  1  (TTSHS  Standing  Assumptions).  The  following  is  assumed  for  the  TTSHSs  con¬ 
sidered  throughout  this  paper: 

A1  The  vector  fields  x  /^(x),  q  e  Q  are  locally  Lipschitz. 

A2  P(h^  =  0)  =  0  for  every  £  e  k  e  he.,  none  of  the  reset-time  distributions  /i^,  £  e 
q  E  Q  have  a  point  mass  at  0. 

A3  P(h^^  =  h^^)  =  0  for  every  £i  £2^  k  e  he.,  for  every  q  e  Q  and  £i  ^  £2  the  reset-time 
distributions  cannot  have  common  point  masses.  □ 


Figure  4:  Feedback  control  loop  with  two  remote  sensors  that  communicate  with  the  controller 
through  a  shared  network  in  Example  1. 


Example  1  (Estimation  through  a  packet-switched  network).  Figure  4  depicts  a  networked  control 
system  that  could  be  modeled  using  a  TTSHS.  It  corresponds  to  a  feedback  configuration  in  which 
the  process  output  vector  y  can  be  decomposed  into  two  components  yi  and  ^2,  each  measured  by 
a  separate  sensor.  The  two  sensors  are  located  far  from  the  controller  and  communicate  with  it 
through  a  shared  network. 

In  this  example,  we  assume  that  the  sensors  take  turns  in  sampling  the  output  and  sending 
their  measurements  to  the  controller.  Specifically,  the  process  output  y  is  sampled  at  times 
k  E  Z^o  and  sensor  1  sends  the  value  of  yi  at  times  tk  with  k  odd  whereas  sensor  2  sends  the  value 
of  y2  at  times  with  k  even.  Between  sampling  times,  the  controller  simply  holds  the  values  of  the 
measurements  received.  This  leads  to  a  discrepancy  between  the  value  of  the  process  output  y{t) 
and  the  input  to  the  controller  y{t)^  at  each  time  t.  Assuming  linear  time-invariant  (LTI)  models 
for  the  process  and  the  controller  of  the  form: 


xp  =  Apxp  -\-  Bpu^ 


yi 

cr 

y2_ 

.C’2_ 

Xp, 


the  system  dynamics  can  be  written  as 


xc  =  Acxc  +  [Bi 


u  =  Ccxc, 


X  =  Ax, 


where 


Xp 

Ap 

BpCc 

0 

0  ■ 

XC 

,  ^:= 

0 

Ac 

Bi 

B2 

yi 

0 

0 

0 

0 

0 

0 

0 

0 

(5) 


in  any  interval  [tkAk+i)^  k  e  Z^q  between  sampling  times;  and  at  each  sampling  time  we 

have 


x{tk)  ’= 


Jodd^ 

«^even  ^  (tfc) 


k  odd 
k  even. 


Jodd 


■  I 

0 

0 

o' 

■  I 

0 

0 

o' 

0 

I 

0 

0 

5  J even 

0 

I 

0 

0 

Cl 

0 

0 

0 

0 

0 

I 

0 

0 

0 

0 

I 

02 

0 

0 

0 

(6) 
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which  expresses  the  fact  that  xp  and  xc  are  continuous  signals  and,  at  odd  sampling  times,  yi{tk) 
is  updated  to  the  value  of  yi{tk)  =  Cixp{tk)^  whereas  y2{tk)  remains  with  the  same  value.  The 
roles  are  reversed  at  the  even  sampling  times. 

Typically,  the  sensors  would  take  measurements  in  a  periodic  fashion,  e.g.,  with  tkpi  —  tk 
always  equal  to  some  (deterministic)  constant  Tg.  However,  when  data  is  sent  through  a  shared 
network,  the  sensors  are  unable  to  send  data  if  the  network  is  busy  and  may  need  to  wait  until 
data  can  be  transmitted.  Assuming  that,  when  the  shared  network  becomes  available  at  time  tk^ 
the  appropriate  sensor  sends  the  current  output  yi{tk)  or  y2{tk)  (depending  on  wether  k  is  odd  or 
even,  respectively),  the  interval  :=  tkpi  —  becomes  a  random  variable  due  to  the  stochastic 
nature  of  the  external  network  traffic  and  the  network  protocols.  Note  that  in  the  case  considered 
here,  one  would  expect  the  distribution  of  tk  to  have  support  in  the  interval  [r5,oo)  and  a  point 
mass  at  Tg  with  weight  equal  to  the  probability  that  the  network  is  available  at  some  fixed  point 
in  time. 

This  networked  control  system  can  be  naturally  modeled  using  the  TTSHS  in  Figure  5.  The 
system  has  two  discrete  states  (modes)  that  keep  track  of  which  sensor  should  send  its  data.  Both 
modes  have  the  same  vector  field  (5),  but  different  reset  maps  defined  by  the  matrices  Jodd  ^md 
Jeven  (6).  The  reset  maps 

01  (^,x)  =  (2,e/odd^)5  02(^7^)  ~  (1?  «>^even^) 7 

are  associated  with  the  reset-time  distributions  /ij(*)  and  intervals  hk  ’=  tkpi  —  tk  for 

k  odd  and  even,  respectively.  This  model  implicitly  assumes  that  consecutive  hk  are  statistically 


Figure  5:  Time- Triggered  Stochastic  Hybrid  System  (TTSHS)  that  models  the  networked  control 
system  in  Example  1. 

independent,  which  means  that  the  delay  experienced  by  one  sensor  before  sending  the  measure¬ 
ment  at  time  tk  is  independent  of  the  delay  experienced  by  the  other  sensor  before  sending  the 
measurement  at  time  tkpi-  This  may  not  be  very  realistic  since  a  large  delay  could  mean  that  the 
network  is  in  a  heavy-load  condition  and  so  subsequent  delays  would  likely  be  high.  However,  this 
modeling  problem  could  be  avoided  by  considering  more  discrete  modes.  E.g.,  one  could  imagine 
that  the  network  has  2  (or  more)  “traffic  modes”  and  switches  stochastically  between  these  modes: 
one  mode  could  correspond  to  low  traffic  and  another  to  high  traffic.  This  would  lead  to  a  TTSHS 
with  4  discrete  states:  k  even  or  odd  and,  for  each  option,  the  two  network  states.  Packet  drops 
could  also  be  considered  using  an  approach  similar  to  that  used  in  [2,  Section  V].  □ 

2.3  State-Driven  Stochastic  Hybrid  Systems 

In  the  TTSHS  model  described  in  Section  2.2,  the  reset  times  t^^  are  random  variables  whose 
distributions  do  not  depend  on  the  continuous  state  x(t).  This  is  not  to  say  that  the  tk  and  x(t) 
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are  statistically  independent,  in  fact,  they  typically  are  not.  However,  while  the  reset  time  affects 
the  construction  of  x(t),  the  distribution  of  the  next  reset  time  >  t  is  allowed  to  depend 

on  the  current  discrete  state  q(t),  but  not  on  the  current  continuous  state  x(t).  In  State-Driven 
Stoehastie  Hybrid  Systems  (SDSHS)  this  is  not  the  case  and  the  continuous  state  x(t)  of  a  SDSHS 
can  (causally)  affect  the  reset  times  by  modulating  the  “instantaneous”  probability  of  resets. 

The  model  of  a  SDSHS  also  requires  three  ingredients: 

1.  a  family  of  (loeally  Lipsehitz)  veet or  fields  {fq  :  ^  e  Q}  that  defines  the  evolution 

of  the  continuous  state  x{t)  in  each  mode  q  e  Q] 

2.  a  family  of  reset  maps  {(fi  :  Q  x  Q  x  e  £}  that  characterizes  how  the  jumps  or 

impulses  change  the  system’s  state;  and 

3.  a  family  of  transition  intensities  {A^  :  Q  x  ^  [0,  oo),^  e  that  defines  when  jumps  or 
impulses  take  place. 

Also  here,  the  families  of  reset  maps  and  transition  intensities  are  parameterized  by  the  same 
(typically  finite)  set  C  so  that,  for  each  I  e  the  transition  intensity  \i  is  associated  with  the  reset 
map  (fe.  Figure  6  shows  a  graphical  representation  of  a  SDSHS:  The  key  difference  with  respect  to 


Figure  6:  State-Driven  Stochastic  Hybrid  System  (TTSHS) 

the  HS  in  Figure  2  is  that  now  the  guard  conditions  have  been  replaced  by  the  transition  intensities 
\l  that  determine  the  “instantaneous”  probability  of  a  reset.  Specifically,  given  the  SDSHS’s  state 
q(t),  x(t)  at  a  time  t  ^  0,  the  probability  that  the  state  will  be  reset  by  (fe  during  the  “elementary” 
interval  (t,t  -h  dt\  is  given  by  A^(q(t), x(t))(it.  More  specifically, 

{1  —  A^(q(t),  x(t))(it  m  =  0, 

A^(q(t),x(t))(it  m  =  1,  (as  dt  j  0^). 

0  m  >  1, 

By  an  “elementary”  time  interval,  we  mean  that  these  equalities  hold  as  the  length  of  the  interval 
dt  converges  to  zero.  One  can  also  think  of  A^(q(t),  x(t))  as  the  “instantaneous”  average  number 
of  resets  by  (fe  per  unit  of  time. 

The  construction  of  a  solution  to  this  SDSHS,  starting  at  an  initial  condition  q{0)  =  qo  e 
x(0)  =  xq  e  M^,  is  analogous  to  the  one  described  in  Section  2.1  for  a  DHS,  but  with  a  new 
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stochastic  mechanism  used  to  generate  the  reset  times.  Specifically,  the  steps  2-5  should  now  be 
replaced  by: 

2”.  Solve  the  initial-value  problem  z  =  /q(^^)(z),  Mt  ^  and  let  denote 

its  maximum  interval  of  existence. 

3”.  Draw  independent  random  numbers  h^,  ^  e  £  with  standard  exponential  distributions  (unit 
mean  and  variance). 

4” .  If  one  of  the  integrals 

“iW  ==  f  •^^(q(tfc),z(s))ds,  Vt  e  [tfc,T),  £  e /:  (7) 

■Jtk 

reaches  h^,  £  e  £  in  the  interval  (tyjj;,T),  i.e.,  if 

tfc+i  :=  inf  {t  e  (tfc,  T)  :3£e  C,  ^  hf} 

belongs  to  the  (open)  interval  {tk,T)  and  3£k  6  C  such  that  ^  then  define  the 

SDSHS  state  in  the  interval  [tk^tk+i]  according  to  (2)-(3).  Increment  the  integer  k  and  go 
back  to  2” . 

5”.  Otherwise,  define  the  SDSHS  state  in  the  interval  [t/;,,T)  according  to  (4)  and  terminate. 

Here  again,  the  random  times  are  called  reset  times  and  this  solution  is  defined  up  to  a  maximal 
time  Tmax  that  can  be  either  equal  to  T  if  the  construction  terminate  through  5”  or  sup^  tk  if  it 
does  not  terminate. 

The  following  conditions  can  be  used  to  make  sure  that  this  construction  succeeds  with  proba¬ 
bility  1  and  will  be  assumed  throughout  this  paper. 

Assumption  2  (SDSHS  Standing  Assumptions).  The  following  is  assumed  for  the  SDSHSs  con¬ 
sidered  throughout  this  paper: 

A4  The  vector  fields  x  ^  fq{x)^  q  e  Q  are  locally  Lipschitz. 

A5  The  transition  intensity  maps  x  ^  A^(g,x),  i  e  q  e  Q  are  continuous  (or  at  least  measur¬ 
able)  . 

A6  The  transition  intensities  A^(g,  x)^  i  e  C  and  the  reset  maps  (l)i{q^  x)^  i  e  C  are  globally  bounded 
in  q  and  locally  bounded  in  x,  in  the  sense  that,  for  every  bounded  set  B  cz  there  exists 
constants  Lq^  Fq  such  that 

\e{q,x)^Lo,  \\(l)i{q,x)\\  ^  Fq,  ,^qe  Q,xeB.  □ 

Assumption  A4  makes  sure  that  the  initial-value  problem  in  step  2”  has  a  (local)  solution  and 
A5  makes  sure  that  the  integrals  in  (7)  are  well  defined.  Assumption  A6  is  not  required  for  the 
construction  of  the  solution,  but  it  will  be  instrumental  in  establishing  global  existence  of  solutions. 
Specifically,  A6  together  with  A4  allow  us  to  conclude  that,  if  the  state  x  remains  inside  a  bounded 
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set,  then  T^ax  =  Consequently,  T^ax  <  QO  must  mean  that,  in  the  interval  [0,Tmax),  x{t) 
leaves  any  bounded  set. 

Finally,  note  that  under  this  construction,  the  h^,  £  e  £  are  all  positive  with  probability  one 
and  one  does  not  have  to  worry  about  tk+i  in  step  4”  being  equal  to  The  probability  of  multiple 
reaching  the  simultaneously  is  also  zero,  so  £k  in  4”  is  well  defined  with  probability  one.  In 
view  of  this,  we  do  not  need  to  worry  about  this  construction  failing  to  produce  cadlag  signals  for 
X  and  q,  which  was  a  concern  in  the  constructions  in  Sections  2.1  and  2.2. 

Example  2  (Estimation  through  a  packet-switched  network).  Figure  7  depicts  a  scenario  where 
one  wants  to  construct  a  remote  estimate  for  the  state  of  a  process.  This  state  is  measured  by 
sensors  that  transmit  their  measurements  to  a  remote  location  through  a  packet-switched  shared 
network.  Our  goal  is  to  accurately  reconstruct  the  state  of  the  process  by  sending  a  relatively  small 
number  of  messages  through  the  network.  Inspired  by  [5,  9,  18,  26,  28],  we  propose  a  stochastic 
rule  to  send  data  that  increases  the  rate  at  which  data  is  sent  as  the  state  estimation  error  grows. 

process  state-estimator 


Figure  7:  Estimation  through  a  packet-switched  network  in  Example  2. 

Figure  7  shows  the  process  to  be  an  LTI  system  driven  by  a  white  noise  process  of  the  form 

X  =  Ax  -h  6w,  X  G  M^,  (8) 

where  w  denotes  a  scalar  Brownian  motion  process.  However,  in  this  paper  we  want  to  focus 
our  attention  on  SHSs  whose  dynamics  in  each  mode  are  given  by  Ordinary  Differential  Equations 
(ODEs)  and  not  by  Stochastic  Differential  Equations  (SDEs),  since  the  former  are  somewhat  simpler 

^  If  the  state  x  remains  inside  a  bounded  set  B,  then  A4  precludes  the  existence  of  finite  escape  time.  In  addition, 
A6  precludes  the  reset  times  to  have  a  finite  accumulation  points.  Indeed,  to  have  t^+i  hnite,  we  must  have 

ml(tfc+i)  =  f  A^(q(tfc),x(s))ds  ^  hi 

for  some  £  E  C.  But  if  (q(t/c),  x(s))  is  upper  bounded  by  Lo,  this  implies  that  Lo{tk+i  —  t/c)  ^  h^.  Having  a  finite 
accumulation  point  for  the  would  thus  require 

^  00  00  00 

y—  y  hfc  ^  y  (tfc+1  -  tfc)  <  CO  ^  y  hi  <  CO, 

k  —  0 

which  is  a  zero  probability  event  since  the  hi  are  i.i.d.  random  variables  with  standard  exponential  distributions. 
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from  a  technical  perspective.  We  will  thus  employ  the  mathematical  trick  of  approximating  an  SDE 
by  a  random  walk  that  can  be  easily  modeled  by  an  ODE  with  stochastic  impulses:  It  turns  out 
that  the  solutions  to  (8)  can  be  obtained  as  the  limit  6  j  0^  of  the  solutions  to  the  SDSHS  of  a 
SHS  with  continuous  dynamics 

X  =  Ax 


and  two  reset  maps 

01  (x)  :=  X  02 (^)  •=  X  —  Vx  G  (9) 

both  with  the  same  transition  intensities 

Xl{x)  =  X2{x)  :=  \/x  G 

It  is  often  the  case  that  this  SDSHS  model  is  close  to  the  physical  mechanisms  that  are  used  to  justify 
the  idealized  white  noise  process.  While  the  idealized  white  noise  process  is  often  mathematically 
very  convenient,  we  shall  see  shortly  that  this  non-idealized  noise  process  can  still  be  simple  to 
analyze. 

The  sensor/encoder  in  Figure  7  samples  the  state  of  the  system  at  random  times  k  e 
and  sends  the  sampled  states  to  a  decoder  that  builds  the  remote  estimate  x  of  the  true  state 

X.  Between  sample  times,  the  decoder  uses  an  open-loop  state  estimator  of  the  form 

X  =  ^x,  Vf  e  [tfc,tfc+i), 

and,  whenever  a  sample  x(tfc)  is  received,  it  resets  the  estimate  x(tfc)  to  the  value  received.  We 
consider  here  a  scenario  where  the  encoder  generates  the  sample  times  tk  stochastically  in  a  state- 
driven  fashion.  Specifically,  we  assume  that  these  times  are  generated  by  a  SDSHS  model  with  a 
transition  intensity  that  is  a  function  of  the  state  estimation  error  e  :=  x  — x.  This  estimation  error 
is  available  to  the  encoder  because  it  can  locally  replicate  the  equations  that  the  decoder  uses  to 
construct  the  estimate  x  and  it  has  direct  access  to  the  true  state  x.  Making  the  generation  of 
sampling  times  that  depend  on  the  estimation  error  makes  intuitive  sense  since  one  would  expect 
that  when  the  state  estimation  error  at  the  decoder  is  very  small,  there  is  little  gain  in  sending  a 
state  measurement.  However,  when  the  error  is  large,  it  is  important  to  correct  this  error  as  fast 
as  possible  by  sending  a  measurement.  We  shall  see  later  that  this  intuition  is  indeed  formally 
justifiable. 

The  error  dynamics  for  the  system  described  above  can  be  summarized  as  follows:  Between 
“event”  times,  the  error  evolves  according  to 

e  =  Ae. 

However,  this  dynamics  is  perturbed  by  two  types  of  events:  jumps  in  x  due  to  the  noise  process 
(9)  that  lead  to  resets  of  the  form 

e(tfc)  =  0i(e“(tfc))  =  e“(tfc)  +  ^/eb,  e{tk)  =  (t>2{e~{tk))  =  e~{tk)  -  Veb, 

both  with  intensity  l/2e;  and  resets  of  the  estimate  x  due  to  the  update  of  the  estimate  that  leads 
to  resets  of  the  form 


e(tfc)  =  ^3(e  (tfc))  :=  0, 


(10) 


14 


with  an  intensity  that  typically  depends  on  the  current  state  estimation  error  e.  These  error 
dynamics  can  thus  be  represented  by  the  SDSHS  in  Figure  8.  This  system  has  a  single  discrete 
mode  and  three  reset  maps,  two  to  model  the  noise  process  and  a  third  one  to  model  the  sampling 
of  the  state. 


Figure  8:  State-Driven  Stochastic  Hybrid  System  (SDSHS)  that  models  the  networked  control 
system  depicted  in  Example  2. 


The  system  depicted  in  Figure  7  and  consequently  the  model  in  Figure  8,  make  several  restrictive 
simplification:  we  assume  full  state  measurements  for  the  encoder,  measurement  noise  is  ignored, 
quantization  of  the  messages  sent  by  the  encoder  is  ignored,  and  it  is  assumed  that  the  network 
does  not  introduce  any  delay.  These  simplifications  are  not  particularly  critical  for  the  analysis 
that  will  follow  and  were  made  mostly  for  simplicity.  From  now  let  us  briefly  mention  that  if  the 
encoder  did  not  have  access  to  the  full  state  or  if  there  were  measurement  errors,  the  encoder  could 
incorporate  a  Kalman  filter  and  send  to  the  decoder,  not  the  state  of  the  process,  but  the  state  of 
the  Kalman  filter  which  “optimally”  encodes  all  the  information  available  up  to  the  current  time. 
Quantization  of  the  values  sent  by  the  encoder  essentially  leads  to  replacing  the  reset  to  zero  of  the 
estimation  error  in  (10)  to  a  reset  of  the  estimation  error  to  the  value  of  the  quantization  error. 
Finally,  one  could  model  the  network  delays  by  adding  an  additional  state  variable  that  keeps  track 
of  messages  that  are  “in  route”  from  encoder  to  decoder  and  make  a  distinction  between  events 
corresponding  to  the  times  at  which  the  state  is  sampled  and  the  times  at  which  the  corresponding 
message  arrives  at  the  decoder.  □ 

2.4  Time- Triggered  Stochastic  Hybrid  Systems  as  State-Driven  Stochastic  Hy¬ 

brid  Systems 

While  TTSHSs  may  seem  to  be  fundamentally  different  from  SDSHS,  TTSHSs  can  be  emulated 
by  appropriately  constructed  SDSHSs  when  the  reset-time  distributions  have  probability  density 
functions,  i.e.,  when  there  exist  functions  :  [0,  oo)  ^  [0,  oo)  such  that 

Fg{h)  ■■=  f  =  f  fgis)ds,  Vh  e  [0,  T^),q  eQJeC, 

Jo 

where  •■=  sup{/i  e  M  :  F^{h)  <  1}  e  M  u  {oo}.  In  this  case,  we  can  construct  a  SDSHS  that 
emulates  the  TTSHS  using  the  following  procedure:  We  start  by  augmenting  the  continuous  state 
of  the  TTSHS  with  a  timer  variable 


T{t)  =  t  —  tk,  Vf  ^  0, 


(11) 
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leading  to  a  SDSHS  with  continuous  state  (x,  r)  e  x  M.  The  vector  field  for  the  state  r  is  simply 

T  =  1, 


and  the  SDSHS  reset  maps  (/)^:QxM^xM^QxM^xM  should  enforce  a  reset  of  r  to  0  at  each 
reset  time  which  leads  precisely  to  (11).  The  SDSHS  should  then  have  transitions  intensities 
A^(g,x,T)  given  by  the  hazard  rate  of  the  reset-time  distribution 


Xl{q,x,T) 


V(?e  re  [0,r^^), 


which  essentially  provides  the  probability  that  a  random  timer  r  with  distribution  /i^  will  fire  in 
an  elementary  interval  [r,  r  -h  dr),  given  that  it  has  not  yet  fired  at  time  r: 


lim 

(ir^O 


P„£  (r  ^  T  <  T  -h  dr  I  T  >  r) 

/Xg  V  I  / 

dr 


4(r) 


To  understand  why  this  works,  we  go  back  to  the  construction  of  the  solution  to  the  TTSHS  in 
Section  2.2,  in  which  the  reset  time  is  determined  by 


tfc+i  =  +  mm 


h 


i 

k 


where  the  are  independent  random  variables  with  distributions  Therefore,  the  (condi¬ 

tional)  distribution  of  tkri  given  and  c[{tk)  is  determined  by 


P  (tfc+i  >  T  I  q(tjE^))  =  P  (h^  >  T  -  Mle  C  \  tk,  q(tjEc)) 


(12) 


In  contrast,  in  a  SDSHS  the  reset  time  is  determined  by 

tfc+i  :=  inf  e  [tfc,T)  :  e  £,  A^(q(tfc),  z(s))(is  ^  h|| 


where  where  z(s)  denotes  the  solution  of  the  continuous  vector  field  initialized  at  the  value  of  the 
state  at  time  tk>>  and  the  h^,  ^  e  £  are  independent  random  variables  with  standard  exponen¬ 
tial  distributions.  Therefore,  the  (conditional)  distribution  of  tk+i  given  tk^  q(tAi)5  is  now 

determined  by 


P  (tfc+i  >  T  I  tfc,  q(tfe),  x(tfc))  =  P  (  A^(q(tfc),  z(5))ds  <  h^,  V£  e  C  |  q(tt),  x(tfc)^ 


ieC 


-£  (q(tfc),z(s))<is 


(13) 


The  above  construction  for  the  SDSHS  guarantees  that 


A^(q(tfc),z(tfc  +  r)) 


Vr  e 
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and  therefore 


rT  rT-tk  rT-tk  — 

A^(q(tfc),z(s))c?s  =  -  A£(q(tfc),  z(tfc  +  r))c?r  =  - — ‘ 

Jtu  Jo  Jo  -1-  — 


q(t/e) 


(^) 


dr 


'  q(t/c) 


(r)' 


nT — 


dr  = 


[log  (l 


q(tfc)' 


T—tk 


(r))  =log(l 


q(tfc) 


(T)), 


which  shows  a  match  between  the  two  conditions  (12)  and  (13)  that  determine  the  resets  of  the 
TTSHS  and  the  SDSHS,  respectively. 

This  ability  to  represent  TTSHS  as  SDSHS,  enables  one  to  combine  in  the  same  SHS  both 
time-triggered  and  state-driven  resets.  However,  this  can  only  be  done,  as  shown  here,  when  the 
reset-time  distributions  have  probability  density  functions  and  therefore  do  not  have  point  masses. 


3  Stability  of  SHSs 

To  characterize  the  stability  of  a  stochastic  process  one  may  pursue  two  alternative  types  of  condi¬ 
tions:  One  is  focused  on  making  sure  that  the  probability  that  the  stochastic  process  “misbehaves” 
is  very  small.  Such  notions  are  often  called  sample-path  stability  notions  because  they  address  the 
behavior  of  individual  sample  paths  of  the  stochastic  process.  Alternatively,  one  may  focus  ones 
attention  on  aggregate  behavior  and  require  that  the  process  “behaves”  well  on  the  average.  Such 
notions  are  called  mean  or  aggregate  notions  of  stability. 

We  start  by  considering  sample  path  notions  of  stability  that  attempt  to  parallel  the  key  concepts 
behind  Lyapunov  stability  for  deterministic  systems.  Sample-path  stability  notions  are  typically 
qualified  by  the  suffix  “in  probability.”  A  SHS  (either  time-triggered  or  state-driven)  with  discrete 
state  q  :  [0,  T^ax)  ^  Q  and  continuous  state  x  :  [0,  T^ax)  ^  is  said  to  be 

D1  (Lyapunov)  stable  in  probability  if  it  is  globally  defined  with  probability  one,  i.e.,  T^ax  ^ 
and,  for  every  e,  p  >  0,  there  exists  a  5  >  0  such  that,  for  every  initial  condition  q(0)  =  qo  e 
x(0)  =  Xq  E  M^, 

ll^oll  ^  ^  P  (^t  E  [0,oo)  :  ||x(t)||  >  ^  6;  (14) 

D2  (Lyapunov)  asymptotieally  stable  in  probability  if  it  is  (Lyapunov)  stable  in  probability  and 
x(t)  converges  to  zero  with  probability  one,  i.e., 

p(^limx(t)-0)  =1. 


Aggregate  or  mean  notions  of  stability  require  the  selection  of  a  non- negative  function  W  :  Q  ^ 
[0,  oo)  and  the  SHS  is  said  to  be 

D3  mean-W  stable  if  it  is  globally  defined  with  probability  one,  i.e.,  T^ax  ^  and,  for  every 
initial  condition  q(0)  =  qo  e  x(0)  =  xq  eMJ^^ 


E 


PF(q(f),x(t)) 


<00,  yt  ^  0; 
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D4  stochastically  mean-W  stable  if  it  is  globally  defined  with  probability  one,  i.e.,  T^ax  =  ^ 
and,  for  every  initial  condition  q(0)  =  qq  e  x(0)  =  xq 


E 


W^(q(t),x(t)) 


dt  <  00; 


D5  asymptotically  mean-W  stable  if  it  is  globally  defined  with  probability  one,  i.e.,  T^ax  =  QO 
and,  for  every  initial  condition  q(0)  =  qq  e  x(0)  =  xq 


lim  E 

t^GC 


W{q{t),x{t)) 


=  0; 


D6  exponentially  mean-W  stable  if  it  is  globally  defined  with  probability  one,  i.e.,  T^ax  =  QO 
and  there  exist  constants  c,  A  >  0  such  that,  for  every  initial  condition  q(0)  =  qo  e 
x(0)  =  Xq  E 


E 


Vt  ^  0. 


When  the  function  W  is  the  squared  Euclidean  norm  of  the  continuous  state  (i.e.,  W{q^x)  =  ||x|p, 
^q^  Q-)  X  E  W^)  we  simply  say  that  the  processes  are  mean-square  stable^  without  reference  to  W. 


4  Analysis  of  Time- Triggered  SHSs 


While  the  stability  analysis  of  general  SHSs  is  challenging,  very  complete  results  appeared  recently 
for  the  class  of  linear  TTSHSs.  These  are  TTSHSs  for  which  the  vector  fields  and  reset  maps  are 
both  linear  functions  of  the  state.  To  present  these  results,  we  consider  the  simpler  case  of  the 
impulsive  system  in  Eigure  9,  with  a  single  linear  vector  field 

X  =  Ax,  (15) 


and  a  single  linear  reset  map 


^{x)  :=  Jx,  Vx  G 

associated  with  a  reset-time  distribution  pi  for  the  inter- reset  intervals  \ik  •=  ^k+i  ~ 


(16) 


Eigure  9:  Linear  Time- Triggered  Stochastic  Impulsive  System 


To  study  the  stability  of  the  linear  time-triggered  impulsive  systems  (TTSIS)  in  Eigure  9,  it  is 
convenient  to  define  a  (stochastic)  discrete-time  system  obtained  by  sampling  the  state  x  at  the 
reset  times  tk'- 


Xfc  :=  x(tfc),  v/c  6  Z^o- 


(17) 
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The  dynamics  of  this  discrete-time  system  can  be  easily  derived  and  are  given  by 

Xfc+i  =  Vfc  e  Z^o, 

and  therefore,  for  a  given  positive  definite  matrix  P  =  P'  >  0,  we  can  compute 

E  [x^+iPxfc+i  I  xfc]  =  xfc  E  (18) 

where  we  used  the  fact  that  value  of  the  state  at  time  is  independent  of  the  interval  hj^  := 
tk^i  —  tk^  Vfc  G  Z^o-  Suppose  now  that  we  can  find  a  positive  definite  matrix  P  for  which 

rcc 

E^e^'hfcj'pjgAhfcj  _  ^  7P,  (19) 

Jo 

for  some  7  g  (0, 1),  where  the  expected  value  is  taken  with  respect  to  the  random  variable  hk  whose 
distribution  is  defined  by  the  reset-time  distribution  /i.  In  this  case,  we  conclude  from  (18)  and 
(19)  that 


E  [xfc+i-Exfc+i  I  Xfc]  ^  7XfcPxfc. 

Taking  expected  values  on  both  sides  of  the  above  inequality,  we  would  then  conclude  that 

E  [x7iPxfc+i]  ^  7E  [x'fcPxfc], 

from  which  it  would  be  straightforward  to  show  that  E[||x)[^|p]  converges  (exponentially  fast)  to  zero 
as  A;  ^  00.  It  turns  out  that  the  stability  of  the  TTSIS  does  not  always  follow  from  this  because 
of  the  behavior  of  x(t)  between  reset  times.  In  particular,  problems  could  arise  if  (i)  A  is  unstable, 
and  therefore  x(t)  tends  to  grow  exponentially  between  reset  times;  and  (ii)  the  distribution  of  hk 
has  infinite  support,  and  therefore  the  interval  between  reset  times  can  be  arbitrarily  large  (albeit 
with  low  probability).  Nevertheless,  by  adding  appropriate  conditions  to  address  the  inter-reset 
behavior  of  x,  we  can  express  fairly  simple  necessary  and  sufficient  condition  for  the  stability  of  the 
TTSIS.  It  turns  out  that  necessary  and  sufficient  conditions  for  the  stability  of  a  TTSIS  depend  on 
the  type  of  stochastic  stability  considered,  as  expressed  by  the  result  below. 

Theorem  1  (Stability  of  a  Linear  TTSIS  [2]).  Consider  a  TTSIS  defined  by  the  veet or  field  (15); 
the  reset  map  (16);  and  the  reset-time  distribution  ji,  for  whieh  Assumption  1  holds.  Further  assume 
that 


a(2/x({6i})  <  r  3A  >  A*  :  J  fi{dh)  <  ao, 

where  a{M)  denotes  the  speetral  radius  of  the  matrix  M ,  the  summation  inside  (j(-)  is  taken  over 
any  point  masses  bi  of  the  reset-time  distribution  pi,  and  A*  denotes  the  real  part  of  the  eigenvalue 
of  A  with  largest  real  part  for  whieh  one  of  the  eorresponding  generalized  eigenveetors  is  not  in  the 
kernel  of  J,  i.e., 

A*  :=  max{3?[A]  :  {A-\lfx  =  Q,  #  0,  e  Z^i,  a;  e  A  e  C}. 
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PI  This  system  is  stochastically  mean-square  stable  (Definition  D4  with  squared  norm  W )  if  and 
only  if  there  exists  a  positive  definite  matrix  P  =  P'  >  0  such  that 


L{P)  <  P, 


poo 

L{P)  ■■=  e^'^J'PJe^^nidh), 

Jo 


and 


poo 

Jo 


(20) 


(21) 


where  X^{A)  denotes  the  real  part  of  the  eigenvalue  of  A  with  largest  real  part,  the 
dimension  of  the  corresponding  Jordan  block,  and  r{h)  :=  P{hj^  >  h)  =  denotes 

the  survivor  function  of  /jl. 


P2  This  system  is  asymptotically  mean-square  stable  (Definition  D5  with  squared  norm  W )  if 
and  only  if  there  exists  a  positive  definite  matrix  P  =  P'  >  0  such  that  (20)  holds  and 

lim  _  Q_  ^22) 

h^oo 


P3  This  system  is  exponentially  mean-square  stable  (Definition  D6  with  squared  norm  W )  if  and 
only  if  there  exists  a  positive  definite  matrix  P  =  P'  >  0  such  that  (20)  holds  and  there  exist 
constants  c,a>0  such  that 


(23) 

□ 


We  can  see  from  Theorem  1  that  the  existence  of  the  positive  definite  matrix  P  =  P'  >  0  such 
that  (20)  holds  is  necessary  for  all  three  types  of  stability.  The  reader  can  easily  verify  that  this 
condition  is  equivalent  to  the  condition  that  we  encountered  in  (19),  and  that  enabled  us  to  show 
that  the  sampled  state  (17)  converges  to  zero  exponentially  fast  as  A;  ^  oo.  However,  we  can  now 
see  that,  in  general,  this  does  not  suffice  to  prove  stability.  The  conditions  (21),  (22),  and  (23)  are 
needed  to  make  sure  that  the  state  remains  “well-behaved”  between  reset  times.  It  turns  out  that 
all  these  three  conditions  hold  trivially  if  the  matrix  A  is  Hurwitz  or  if  the  distribution  of  the  hj^ 
have  finite  support  (i.e.,  if  3/i  >  0  such  that  r{h)  :=  P(h;j^  >  h)  =  0).  For  unstable  matrices  A  and 
distributions  with  infinite  support,  stability  essentially  depends  on  how  the  rate  of  growth  of 
compares  with  the  survival  function  r{h)  of  the  reset-time  distribution  /i. 

From  a  computational  perspective,  the  condition  (20)  is  a  Linear  Matrix  Inequality  since  the 
map  P  1-^  L{P)  is  linear.  The  existence  of  the  matrix  P  can  thus  be  easily  verified  numerically. 
This  same  condition  can  also  be  verified  by  computing  the  spectral  value  of  an  appropriately  defined 
matrix  or  through  the  evaluation  of  a  Nyquist  contour  [2].  The  conditions  (21),  (22),  and  (23)  are 
also  easy  to  verify  as  they  simply  depend  on  the  “least  stable”  eigenvalue  of  A. 

While  the  sufficiency  of  the  conditions  in  Theorem  1  is  fairly  easy  to  establish  based  on  the 
reasoning  that  we  started  to  outline  above,  their  necessity  requires  a  more  sophisticated  proof  that 
can  be  found  in  [2].  In  addition,  the  reader  is  referred  to  [1]  for  results  pertaining  to  multiple  resets 
and  to  [4]  for  multiple  resets  and  multiple  discrete  modes.  In  these  references,  the  reader  can  also 
find  numerical  examples  illustrating  how  these  results  can  be  used  to  analyze  NCSs  analogous  to 
the  one  described  in  Example  1. 
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5  Analysis  of  State-Driven  SHSs 

Lyapunov-based  methods  provide  a  very  versatile  set  of  tools  that  can  be  used  to  establish  the 
stability  of  (deterministic)  Ordinary  Differential  Equations  (ODEs)  like 

X  =  f{x)^  X  G  (24) 

A  key  enabling  tool  for  Lyapunov-based  methods  is  the  chain-rule  for  taking  derivative,  which 
enable  us  to  compute  the  time-derivative  of  a  function  along  trajectories  of  the  ODE.  Specifically, 
given  a  continuously  differentiable  function  V  :  ^  M,  we  have  that 

dV{xit))  ^  dV{xit))  dxjt)  ^  dV{x{t))  . 

dt  dx  dt  dx  V  /  ’ 

for  t  in  any  interval  [0,T)  on  which  the  solution  to  (24)  is  defined  and  continuously  differentiable. 
This  result  enables  one  to  conclude  that  the  algebraic  condition 

^0,  'ixe  R” 

OX 

suffices  to  conclude  that  the  time  function  t  ^  V {x{t))  is  monotone  non-increasing  along  solutions 
to  (24),  even  if  one  is  unable  to  explieitly  eompute  sueh  solutions.  To  extend  this  type  of  tools  to 
SDSHSs  we  need  something  like  the  chain  rule  for  this  type  of  SHSs.  Fortunately,  such  rule  exists. 


5.1  Extended  Generator 


Consider  the  SDSHS  defined  in  Section  2.3.  We  define  the  extended  generator  of  this  SDSHS  to 
be  the  operator  L  that  maps  a  continuously  differentiable  function  V  :  Q  x  ^  M  into  another 
function  LV  :  Q  x  ^  M  according  to  the  rule: 


LV{q,x) 


SV{q,x) 

dx 


fqO)  +  L  (^  ^ ®))  ’ 

eeC 


^qeQ,  xe  M**. 


It  turns  out  that,  under  appropriate  assumptions,  one  can  use  the  extended  generator  to  compute 
derivatives  along  solutions  to  the  SDSHS.  In  particular. 


E[y(q(f),x(f))  I  q(r),x(r)]  '"= 

^(q('r),x(r))  +  J  E  [LE(q(s),x(s))  I  q(r),x(r)]ds,  Vt  ^  r  ^  0,  (25) 


where  refers  to  the  fact  that  this  equality  holds  with  probability  one.  In  the  Markov  processes’ 
literature,  (25)  is  often  known  as  Dynkin’s  formula.  This  formula  allows  us  to  conclude  from  the  Fun¬ 
damental  Theorem  of  Calculus  for  Lebesgue  Integrals  [12,  p.  102]  that  t  ^E[V  (q(t),  x(t))  I  q(r),x(r)] 
is  absolutely  continuous  and  differentiable  almost  everywhere,  with 


dE  [E(q(t),x(t))  I  q(r),x(r)] 
dt 


'^=°E[Ly(q(t),x(f))  |q(r),x(r)], 


Mt  ^  [r,  oo),  Vr  ^  0, 


(26) 


where  the  g  refers  to  the  fact  that  the  derivative  may  not  exist  on  a  set  of  times  t  with  zero  measure. 
It  is  important  to  note  that,  even  though  the  sample  paths  of  the  stochastic  process  E(q(t),  x(t)) 
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are  typically  discontinuous  because  of  the  resets,  the  expected  value  of  this  stochastic  process  is 
not  only  continuous  but,  in  fact,  differentiable  almost  everywhere.  We  shall  refer  to  (25)  as  the 
differential  form  of  Dynkin^s  formula. 

Either  form  of  the  Dynkin’s  formula  (for  r  =  0)  allow  us  to  conclude  that  an  algebraic  condition 
of  the  form 


LV{q,x)  ^0,  Vge  Q,  x  e  M^, 

suffices  to  establish  that  the  time  function  t  ^  E  [’\/(q(t),  x(t))]  is  upper  bounded  by  E(go?^o) 
with  probability  one,  which  is  the  basis  to  establish  stability  for  SDSHSs,  even  if  one  is  unable  to 
explicitly  compute  its  solutions  or  the  expectations  that  appear  in  (25)  or  (26). 

A  technical  but  important  issue  in  applying  Dynkin’s  formula  is  whether  or  not  (25)  holds  for 
a  particular  function  E  :  Q  x  ^  M.  The  following  result  adapted  from  [11]  provides  sufficient 
conditions  for  the  validity  of  Dynkin’s  formulas. 

Theorem  2.  Consider  the  SDSHS  defined  in  Seetion  2.3  and  a  eontinuously  differentiable  funetion 
E  :  Q  X  ^  M.  Suppose  that  Assumption  2  holds  and,  in  addition,  the  following  eonditions  also 
hold: 


A7 

A8 


A9 


The  differential  equations  z  =  fq{z) ,  q  e  Q  all  have  globally  defined  solutions  for  every  initial 
eondition  2:(0)  =  zq  ^ 

For  every  initial  eondition  q(0)  =  q^)  e  Q,  x(0)  =  xq  e  we  have  that 

E[N(r)]  <00,  vr  ^  0  (27) 


where  the  (cadlag)  stochastic  process  N(r)  :=  max|fc  e  Z^o  :  ^  T}  counts  the  number  of 

resets  up  to  time  T. 


For  every  initial  condition  q(0)  =  qo  ^  Q,  x(0)  =  xq  e  E”,  we  have  that 
rN(T) 


E 


Yi  -v{q  (tfc),x  (tfc)) 


L  k^O 


<  00, 


VT^O. 


(28) 


Then  the  SDSHS  state  z  :=  (q,  x)  is  a  strong  Markov  proeess  for  whieh  (25)  and  (26)  hold.  □ 

One  should  clarify  that  Theorem  2  requires  the  left-hand  sides  of  (27)  and  (28)  to  be  finite  for 
every  initial  condition  and  every  finite  T,  but  it  does  not  require  the  existence  of  a  uniform  bound 
over  all  initial  conditions  and  times  T. 

We  recall  that  a  stochastic  process  z{t)  e  Vt  e  [0,  oo)  is  called  a  Markov  proeess  if,  for  every 
measurable  set  A  ^ 

P  (z(t)  G  A  I  z(s).  Vs  ^  r)  =  P  (z(t)  G  A  I  z(t))  Vt  ^  t  ^  0;  (29) 

which  means  that  the  only  information  relevant  to  evaluating  the  behavior  of  the  process  beyond 
tome  T  is  z(t).  The  same  process  is  a  strong  Markov  proeess  if  (29)  holds  not  just  for  deterministic 
times  t,T,  but  also  for  stochastic  times  t,T  that  are  stopping  times  for  the  natural  filtration  of  z. 
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We  recall  that  a  random  variable  T  is  a  stopping  time  for  the  natural  filtration  of  z  if,  for  every 
t  ^  0,  one  can  determine  whether  or  not  T  ^  t  by  observing  z(5),  \/s  ^  t.  We  shall  encounter 
examples  of  stopping  times  shortly. 

While  essential,  Assumptions  A8-A9  are  often  difficult  to  verify  directly  and  can  be  violated  by 
apparently  innocuous  SDSHSs.  Consider,  e.g.,  a  SDSHS  with  a  single  discrete  mode  and  a  scalar 
state  X  G  M  that  evolves  according  to  x  =  0  between  transitions,  a  single  (nonlocally  Lipschitz) 
reset  map  (/)i{x)  :=  \/x  e  M,  and  a  constant  transition  intensity  Xi{x)  :=  /i,  Vx  g  M  for  constants 

p  ^  1  and  /i  >  0.  Given  a  time  instant  t  >  0,  it  is  straightforward  to  conclude  that  x(t)  =  Xq 
and  therefore 


00  7)^  rp  ,p 

E[x(f)]=  y 

n\ 

It  turns  out  that  when  xq  >  1  the  above  series  does  not  converge  for  any  t  >  0  and  therefore  (25) 
does  not  hold  for  any  t  >  t  :=  0.  This  does  not  contradict  Theorem  2  because  Assumption  A9 
does  not  hold.  Problems  may  still  arise  when  the  reset  maps  are  globally  Lipschitz.  For  example, 
considering  instead  the  globally  Lipschitz  reset  map  (/)i{x)  :=  px,  Vx  g  M  and  a  linear  transition 
intensity  Xi{x)  =  x^Mx  e  M,  it  is  possible  to  show  that  E[x(t)]  explodes  in  finite  time  and  therefore 
(25)  only  holds  on  a  finite  time  interval.  For  this  system,  neither  A8  nor  A9  holds. 


5.2  Stopped  Processes 

As  mentioned  above,  a  key  challenge  in  using  Dynkin’s  formulas  to  prove  stability  of  a  SDSHS  is 
that  often  it  is  not  easy  to  establish  that  the  assumptions  of  Theorem  2  hold.  In  particular,  the 
differential  equations  may  have  finite  escape  time,  which  invalidates  A7;  the  number  of  resets  in 
a  finite  interval  could  potentially  be  unbounded,  which  would  invalidate  A8;  or  the  summation  in 
(28)  could  be  unbounded,  which  would  invalidate  A9.  In  fact,  the  goal  of  proving  stability  of  a 
SDSHS  is  often  precisely  to  establish  that  these  types  of  “misbehavior”  cannot  occur.  A  technique 
that  can  be  used  to  overcome  this  challenge  goes  back  at  least  50  years  and  consists  of  analyzing 
an  auxiliar  “stopped”  process  (see,  e.g.,  [19]),  for  which  it  is  easier  to  establish  the  validity  of  the 
Dynkin’s  formula,  and  from  which  one  can  still  conclude  stability  of  the  SDSHS. 

Given  a  set  (9  ^  Q  x  we  define  the  O-stopping  time  to  be  the  first  time  that  the  SDSHS’s 
state  z{t)  :=  (q(t),  x(t)),  t  e  [0,  Tmax)  leaves  O.  Specifically, 


To 


Tmax  if  z{t)  eO,^te  [0,  T^ax) 

inf  {f  6  [0,  Tmax)  :  z(t)  ^  O]  otherwise. 


(30) 


All  stopping  times  considered  in  this  paper  are  defined  using  sets  O  cz  Q  x  that  are 

AlO  open  in  x  in  the  sense  that,  for  every  q  e  Oq  :=  {x  e  :  (q^x)  g  O}  ^  is  an  open 
subset  of  and 


All  uniformly  hounded  in  x  in  the  sense  that  there  exists  a  constant  Cn  such  that 
V(g,x)G(9. 


X 


^  Co, 
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Because  the  state  z  is  right-continuous  and  q  is  piecewise  constant,  AlO  guarantees  that  the  infimum 
in  the  second  branch  of  (30)  is  actually  a  minimum  and  therefore 

wpo  wpo 

z{t)  e  O,  Vie  [0,  To),  and  z{t)  ^  O,  Vt  e  [To,  T^ax), 

where  the  right-hand  side  condition  only  makes  sense  when  To  <  Tmax-  Also,  as  discussed  in 
Section  2.3,  if  the  state  x  remains  inside  a  bounded  set  and  Assumption  2  holds,  then  T^ax  =  QO. 
Therefore,  for  sets  O  satisfying  All,  we  must  have  P(T(p  =  Tmax,Tmax  <  oo)  =  0.  Under  these 
assumptions,  we  can  define  the  O-stopped  processes  to  be  the  stochastic  processes 

[x(To)  t  G  [To,  oo).  To  <  Tmax,  [q(To)  t  G  [To,  oo).  To  <  Tmax, 

Note  that,  because  P(To  =  T^ax,  T^ax  <  oo)  =  0,  we  do  not  need  to  consider  the  case  t  g  [To,  oc), 
with  To  =  Tmax. 

Aside  from  the  technicalities  mentioned  above,  which  are  needed  to  make  sure  that  the  stopped 
processes  are  well-defined,  the  stopped  processes  (qo,xo)  are  equal  to  the  SDHSH  state  (q,  x) 
until  the  first  time  To  that  the  state  exits  O  and  remain  constant  after  that  time. 


5.2.1  Extended  Generator  of  the  Stopped  Processes 

It  turns  out  that  the  stopped  process  xo  also  satisfies  Dynkin’s  formula,  but  for  an  extended 
generator  that  is  zero  outside  O.  Specifically, 


E  y(qo(t),xci(t))  I  qo(r),xei(r)]  '^=° 

E(qei(r),xe,(r))  +  J  E  [Lol/(qo(s),xe,(s))  I  qe,(r),xe,(r)]ds,  Vt  ^  r  ^  0,  (31) 


and 


dE  [E(qg,(t),xo(t))  I  qg>(r),X0(r)] 

wpo 

dt 

E  [Lc)E(qc)(l),xo(l))  I  qo(r),xc)(r)],  Vt  f  [r,  go),  r  ^  0,  (32) 

where 


LoV{q,x) 


LV{q,x)  {q,x)eO 
0  {q,x)^0. 


(33) 


When  AlO-All  hold,  the  assumptions  needed  for  the  validity  of  the  Dynkin’s  formula  (32)  are 
much  easier  to  verify  than  those  in  Theorem  3,  as  noted  in  the  following  result  also  adapted  from 
[11]: 


Theorem  3.  Consider  the  SDSHS  defined  in  Section  2.3,  a  set  O  cz  Q  xM^  that  satisfies  AlO-All, 
the  associated  O-stopped  processes,  and  a  continuously  differentiable  function  V  :  Q  x  ^  M. 
If  Assumption  2  holds,  then  zq  '=  (qc>,xc))  is  a  strong  Markov  process  for  which  (31)  and  (32) 
hold.  □ 
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5.2.2  Convergence  of  the  Stopped  Processes 

The  key  technical  result  needed  to  use  stopped  processes  to  draw  conclusions  on  the  state  of  the 
SDSHS  is  that,  when  the  solution  to  the  SHSDS  exists  globally  (i.e.,  T^ax  Qo),  the  stopped 
processes  c[o  stnd  converge  (pointwise  in  time)  to  the  states  q  and  x  of  the  SDSHS,  as  we  enlarge 
the  set  O  to  cover  the  whole  Q  x  This  allows  us  to  use  Fatou’s  Lemma  to  establish  bounds 
on  probabilities  or  expected  values  involving  the  SDSHS  state,  from  bounds  involving  the  stopped 
processes. 

Lemma  1.  Consider  the  SDSHS  defined  in  Seetion  2.3,  a  monotone  sequenee  of  sets  {Oi  cz  O2  ^ 

•  •  •  cz  Om  ^  }  that  satisfy  A  10-All  and  for  whieh  Um=i  =  Q  'x  IR’^;  ctnd  the  assoeiated 

Om-stopped  proeesses.  If  Assumption  2  holds  and  that  the  solution  to  the  SDSHS  exists  globally 
with  probability  one  [i.e.,  P(Tmax  =  ^  then: 

P4  For  every  finite  time  t  ^  0, 

P  (  lim  qo^(t)  =  q(t),  lim  xo^(f)  =  x(f))  =  1. 

\  m^oo  m^oo  / 

P5  For  every  finite  time  t  ^  0,  eonstant  c  ^  0,  and  non-negative  funetions  V,  Vm  :  Q  x  ^ 
[0,  oo),  sueh  that  Vm{q^  x)  =  V {q,  x),  V(g,  x)  e  Om^  Vm  e  we  have  that 

Vi(qo^(i),xo^(t))  is 

and  also  that 
llVm{q.Om{s),^oZs))ds 

E  [So^m(qO^(s),XC>^(s) 

(35) 

□ 


measurable 
^  c,  Vm  G 


'F(q(t), x(t))  is  measurable 


E 


E(q(t),x(t)) 


^  c. 


(34) 


is  measurable 


)d6 


^  c,  Vm  G 


^£E(q(s)  ,  x(s))(is  is  measurable 
y  [  £P(q(s),x(s))dsl 


^  c. 


Proof  of  Lemma  1.  Given  an  arbitrary  finite  time  t  ^  0, 
t  <  Tmax  ^  sup  ||x(5)||  <00  ^  3m*  >  0,  Vm  ^  m*  :  Tq^  >  t  ^ 

^  lim  qo^(t)  =  q(t),  lim  xc.^(t)  =  x(t). 

m^oo  m^oo 


Therefore 


lim  qo^ 

rn^OO 


Xt)  =  <iX), 


lim  xe,„ 

n^co 


(t)  =  x(t)  )  ^  P(Tmax  >t)^  P(Tmax  =  OO)  =  1, 


from  which  proposition  P4  follows.  To  prove  proposition  P5,  we  first  use  a  similar  argument  to 
prove  that 

P  (yyoEm(qo^(t),Xo^(t))  =  P(q(f),x(t)))  =  1. 

Since  the  Vm  are  non-negative,  we  can  then  use  Fatou’s  Lemma  to  conclude  that  'F(q(t), x(t))  is 
measurable  and 


E 


P(q(t),x(t)) 


^  lim  inf  E 

m^oo 


<  C, 


from  which  (34)  follows.  A  similar  argument  can  be  used  to  prove  (35). 
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5.3  Lyapunov-based  Analysis 

We  say  that  a  function  V  iQxA’^Misa  candidate  Lyapunov  function  for  the  SDSHS  defined  in 
Section  2.3  if  it  satisfies  the  following  properties: 

A12  For  each  fixed  q  ^  the  function  x  ^  V{q^x)  is  continuously  differentiable. 

A13  V  is  non-negative  in  the  sense  that  V{q^  x)  ^  0,  Vg  e  Q,  x  e 
A14  V  is  radially  unbounded  in  in  the  sense  that,  Vm  >  0,  the  set 

Om  •=  {{q,x)  e  Q  xR^  :V{q,x)  <  m}  (36) 

satisfies  All  (and  also  AlO  because  of  A12). 

The  following  result  can  be  used  to  establish  stability  in  probability  and  is  adapted  from  [21, 
Theorem  2.1,  p.  524],  [19,  Theorem  1,  p.  38]  and  [19,  Theorem  2,  p.  39]. 

Theorem  4  (Stability  of  a  SDSHS).  Consider  the  SDSHS  defined  in  Section  2.3  and  an  associated 


candidate  Lyapunov  function  V  \  Qx  ^  M. 

P6  If 

LV{q,  x)  ^  pV,  Vg  G  Q,  X  G  (37) 

for  some  constant  p  ^  0  then,  for  every  initial  condition  q(0)  =  q^)  e  Q,  x(0)  =  xq  e  the 
solution  to  the  SDSHS  is  globally  defined  with  probability  one,  i.e.,  P(Tmax  =  eo)  =  1. 

P7  If  (37)  holds  with  p  =  0,  i.e., 

LV{q,x)^0,  \/qeQ,xER^  (38) 

then,  for  every  m  >  0  and  every  initial  condition  q(0)  =  q^)  e  Q,  x(0)  =  xq  e 

P  (3te  [0,oo)  :  't/(q(t),x(t))  ^  /x)  X  V{qo^xo)  ^  V/xe(0,m].  (39) 

P8  If,  in  addition  to  (38),  V  is  positive  definite,  in  the  sense  that  there  exist  functions  71,72  of 
class  JC  such  that 

7i(l|a;||)  ^  ^  72(lkll),  yqeQ,xeW^  (40) 

then  the  SDSHS  is  stable  in  probability  (Definition  Dl). 

P9  If,  in  addition  to  (38)  and  (40),  LV  is  negative  definite,  in  the  sense  that  there  exists  a 
function  a  of  class  fC  such  that 

LV{q,  x)  ^  -a{\\x\\),  Vg  G  Q,  x  G 

then  the  SDSHS  is  asymptotically  stable  in  probability  (Definition  D2).  □ 
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In  view  of  (25)  or  (26),  the  condition  (38)  essentially  means  that  the  expected  value  of  V (q(t),  x  («)) 
should  decrease  along  solutions  to  the  SDSHS.  This  non-increase  on  the  average  would  hold  a- 
fortiori  if  we  demanded  that 

— ^  0  and  V{4>(,{q,x))  ^V{q,x),  e  £,  g  e  Q,  a;  e  R”,  (41) 

where  the  left-hand  side  inequality  essentially  requires  y(q(t), x(t))  to  decrease  along  the  (deter¬ 
ministic)  flows  of  X  =  /^(x)  and  the  right-hand  side  inequality  requires  V (q(t),  x(t))  not  to  increase 
at  each  reset  time.  However,  (38)  is  weaker  than  (41)  because,  for  (38)  to  hold  the  decrease  only 
needs  to  occur  in  expected  value,  which  permits  some  sample  paths  to  exhibit  strict  increase  on  the 
value  of  V (q(t),  x(t))  along  flows  of  x  =  /g(x)  and/or  strict  increase  on  the  value  of  V (q(t),  x(tyjj;)) 
at  reset  times. 

The  following  results  can  be  used  to  establish  aggregate  notions  of  stability.  It  assumes  that 
the  solution  to  the  SDSHS  is  globally  defined,  which  could  be  established,  e.g.,  using  proposition 
P6  in  Theorem  4. 


Theorem  5  (Mean  stability  of  a  SDSHS).  Consider  the  SDSHS  defined  in  Section  2.3,  an  associ¬ 
ated  candidate  Lyapunov  function  V  :  Q  x  M,  and  assume  that  the  solution  to  the  SDSHS  is 

globally  defined  with  probability  one,  i.e.,  P(Tmax  =  oo)  =  1. 


PIO  If  (38)  holds,  then  the  SDSHS  is  mean-V  stable  (Definition  D3)  and,  for  every  initial  condi¬ 
tion  q(0)  =  q^E  Q,  x(0)  =  E 

E  E(q(t),x(f))  ^y(go,a:o),  Vf  ^  0. 

and  the  solution  to  the  SDSHS  is  globally  defined  with  probability  one,  i.e.,  P(Tmax  =  oo)  =  1. 

Pll  If 

LV{q,x)  ^  -W{q,x),  \fq  E  Q,xgM^, 

for  a  function  VP  :  Q  x  ^  M  such  that  W{q,x)  ^  0,  \/q  e  Q,x  e  then  the  SDSHS 
is  stochastic  mean-W  stable  (Definition  D4)  and,  for  every  initial  condition  q(0)  =  q^)  e  Q, 
x(0)  =  x^  E 


poo 

E  W{qit),K{t))  ^V{qQ,XQ),  Vt^O. 
Jo  L 


(42) 


P12  If 


LV {q,  x)  ^  —pV {q,  x)  +  c,  'iq  e  Q,x  e  E"", 


(43) 


for  constants  c  ^  0,  p  >  0,  then  the  SDSHS  is  mean-V  stable  (Definition  D3)  and,  for  every 
initial  condition  q(0)  =  q^)  e  Q,  x(0)  =  Xf)  e 


E 


V(q(t),x(t))  ^  e  ^*V(qo,xo)  + -(1  -  e  Vt^O. 


(44) 


In  case  c  =  0,  then  the  SDSHS  is  exponentially  mean-V  stable  (Definition  D6). 
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P13  If 


LV (g,  x)  ^  —W (g,  x)  +  c,  E  Q^x  E 


for  a  constant  c  ^  0  and  a  function  VK  :  Q  x  ^  M  such  that  W (g,  x)  ^  0,  \/q  e  Q^x  e 
then,  for  every  initial  condition  q(0)  =  q^)  e  Q,  x(0)  =  x^)  e 


lim 

T^oo 


^jjE[t^(q(f),x(t)) 


dt  ^  c 


Computationally  effective  systematic  procedures  to  find  Lyapunov  functions  for  SDSHSs  are 
generally  not  known,  but  a  notable  exception  are  the  SDSHSs  considered  in  Section  2.4  to  emulate 
linear  TTSHSs.  We  saw  in  Section  2.4  that  the  TTSIS  defined  by  the  vector  field  (15),  the  reset 
map  (16),  and  a  reset-time  distribution  fi  for  which  there  exists  a  probability  density  function  /, 
i.e., 

F{h)  :=  f  ^{ds)  =  f  f{s)ds,  V/i  G  [0,  T),  T  :=  sup{/i  G  M  :  F{h)  <  1}  G  M  u  {oo},  (45) 

J[0,/i]  Jo 

can  be  emulated  by  a  SDSHS  with  continuous  state  (x,  r)  g  x  M;  a  vector  field  defined  by 

X  =  Ax,  T  =  1]  (46) 

and  a  single  reset  map  (f  and  transition  intensity  A  defined  by 

:=  {JX,0),  X{t)  :=  ^X  E  ,  T  E  [0,T) .  (47) 

1  -  L  (T  j 

Theorem  6.  Consider  a  TTSIS  defined  by  the  vector  field  (15),  the  reset  map  (16),  and  a  reset-time 
distribution  fi,  for  which  Assumption  1  holds.  Further  assume  that  the  reset-time  distribution  pi  can 
be  defined  by  a  probability  density  function,  i.e.,  when  there  exists  a  function  f  :  [0,oo)  ^  [0,oo) 
such  that  (45)  holds.  This  system  is  uniformly  exponentially  mean-square  stable  (Definition  D6 
with  squared  norm  W)  A  and  only  if  there  exists  a  positive  definite  matrix-valued  function  P  : 
[0,  oo)  ^  ]^nxn  constants  ci,  C2  >  0  such  that,  for  every  positive  definite  matrices  S^i,  5^2  e 

Cl/  <  P(r)  <  C2/,  P(t)  =  -5i  -  A'P{r)  -  P{t)A  -  A(t)  ( J'P(O)  J  -  P(t)  +  ^2) ,  Vr  g  [0,  T). 

In  this  case,  V{x,t)  :=  FP{t)x,  \/x  e  ^  0  is  a  candidate  Lyapunov  function  for  which  (43) 

holds  with  c  =  0  for  the  SDSHS  defined  by  (46) -(47).  □ 

Example  2  (Estimation  through  a  packet-switched  network  (cont.)).  As  mentioned  before,  the 
error  dynamics  of  the  state  estimation  error  e  for  the  networked  control  system  considered  in 
Example  2  can  be  modeled  by  the  (single-mode)  SDSHS  in  Figure  8,  whose  extended  generator  is 
of  the  form 

LV{e)  ■■=  +  A(e)(E(0)  -  E(e)) 

+  f^{v{e  +  V~eb)-Vie))  +  f^(v{e-V~eb)-V{e)),  Ve  e  M",  (48) 
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for  a  given  function  V  :  ^  M.  The  first  step  in  analyzing  the  resulting  SDSHS  is  to  establish 

global  existence  of  solution.  To  this  effect  we  consider  a  candidate  Lyapunov  function  of  the  form 

^lobaK^)  ll^lP  +  C,  Ve  G 

for  some  appropriately  chosen  constant  c  ^  0.  Applying  the  extended  generator  (48)  to  IgiobaU 
leads  to 

LVgioi)s^i(e)  =  e! {A  A'  —  A(e))e  +  ^  crmax[A  +  A^]||e|p  + 

=  CJ’maxfA  +  A']  (V^lobaK^)  ~  ^ 

where  crmax[A  +  A']  denotes  the  largest  singular  value  of  A  +  A'.  Picking  c  :=  &^V^max[A  +  A'], 
we  conclude  that  TV^iobai(e)  ^  crmax[A  +  A']V^iobai(e),  Ve  g  and  we  can  use  proposition  P6  in 
Theorem  4  to  conclude  global  existence  of  solution  with  probability  one,  regardless  of  our  choice 
for  the  intensity  A(e). 

To  study  mean  stability  of  the  SDSHS,  we  consider  polynomial  candidate  Lyapunov  functions 
of  the  form 


14(e)  (e'Pe)^,  Ve  g  M^, 

with  a'^  1.  Applying  now  the  extended  generator  (48)  to  14,  leads  to 

LT4(e)  =  a{e' Pe)^-^e' {PA  +  A'P)e  -  A(e)(e'Pe)“ 

((e  +  ■s/ebyP{e  +  \/eb))°‘  +  ((e  —  ■s/ebyP{e  —  Vefe))"  —  2(e'Pe)" 
^  2e 

which  can  be  written  as 


LK,(e)  =  -pV{e)  +  e(e),  Ve  g 
for  any  constant  p  >  0,  provided  that  we  define 

c(e)  :=  a(e'Pe)“-^e'(pa  +  A' P  -  ~^p)e 

((e  +  ^/ebyP{e  +  ^/eb))^  +  ((e  —  ^/ebyP{e  —  Ve6))“  —  2(e'Pe)“ 

^  2^  ■ 

To  check  whether  or  not  e(e)  is  bounded,  we  can  use  the  fact  that,  since  P  is  positive  definite,  the 
dominante  terms  in  e(e)  as  e  ^  oo  are  the  powers  in  (e'Pe)^: 

a(e'Pe)““^e'(p^  +  A'P  -  ^^^^4)® 

which  is  negative  as  long  as 

PA  +  A'P  -  MduTp  <  0. 

In  case  A(e)  is  radially  unbounded^  this  is  always  the  case  for  sufficiently  large  e  and  we  conclude 
that 

LVa{e)  ^  -pV{e)  +  c^ax,  Ve  g  M"",  c^ax  =  sup  c(e)  <  oo. 

ceR'^ 
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Using  proposition  Pll  in  Theorem  5  we  conclude  that  if  the  intensity  A(e)  is  radially  unbounded, 
the  SDSHS  is  mean-Uo,  stable  and  consequently  the  2(a-moment  of  the  error  Efljelp*^]  is  bounded, 
for  every  a  ^  1. 

In  case  A(e)  eventually  “saturates”  for  large  e,  i.e., 

>  0  :  ||e||  >  k.  A(e)  =  Aqo,  Ve  e 


then  mean-V^  stability  can  be  established  if  we  can  find  a  positive  definite  matrix  P  and  a  positive 
constant  p  for  which 

PA  +  A'P  -  =  p(a-  +  (a  -  '^^^i)'p  <  0. 

a  V  2a  J  \  2a  J 

This  turns  out  to  be  the  case  if  and  only  if  all  the  eigenvalues  of  A  have  real  part  strictly  less  than 


where  X^{A)  denotes  the  real  part  of  the  eigenvalue  of  A  with  largest  real  part.  In  case  A  has  no 
eigenvalues  with  positive  real  part,  this  condition  always  holds  and  we  again  can  conclude  that  the 
2(a-moment  of  the  error  Efljelp"^]  is  bounded,  for  every  a  ^  1.  However,  if  A  has  any  eigenvalue 
with  positive  real  part,  the  condition  above  is  equivalent  to 


n 

2Agf^(A)  ’ 


which  means  that  only  some  of  the  moments  E[||e|p^]  may  be  bounded. 


□ 


5.4  Martingales  and  Doob’s  inequality 

While  complete  proofs  of  the  results  summarized  in  Section  5.3  is  beyond  the  scope  of  this  paper, 
it  is  worth  describing  the  key  steps  behind  the  Lyapunov-based  analysis  of  SDSHSs,  as  these  may 
enable  the  reader  to  construct  alternative  results.  These  proofs  rely  heavily  on  the  concept  of 
Martingale  and  the  pioneering  results  by  Doob. 

We  recall  that  a  stochastic  process  M(t),  t  ^  0  is  called  a  supermartingale  for  a  filtration 
{T,  :  t  ^  0}  if 

E[M(t)]  <00,  Vt  ^  0,  E[M(t)  I  3^r]  ^  M(r),  Vt  ^  r  ^  0. 

Intuitively,  this  means  that  the  eurrent  value  of  M(t)  provides  an  upper  bound  on  future  expee- 
tations  of  M(t),  t  ^  r.  Because  a  supermartingale  only  decreases  “on  average,”  specific  sample 
paths  may  occasionally  increase.  However,  the  probability  of  a  large  increase  can  be  upper  bounded 
through  Doob’s  martingale  inequality  [19,  Section  1-7.]: 

Lemma  2  (Doob’s  martingale  inequality).  //M(t),  t  0  is  a  non-negative  supermartingale  for 
the  filtration  t  ^  0},  then 

/  r  N  /  N  \  E  rM(0)] 

P  (3t  G  [0,oo)  :  M(t)  ^ /i)  ^  ^  ^  V/i>0.  □ 
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5.4.1  Proof  of  Theorem  4 

In  Lyapunov-based  stability  proofs,  the  supermartingales  of  interest  are  of  the  form 


MoAt)  ■■=  e-P^V{cioAtUoAt)),  Vf  ^  0, 


(49) 


with  p  ^  0  and  sets  Om  of  the  form  (36).  To  verify  that  this  process  is  a  supermartingale  under 
(37)  in  P6,  we  use  (32)  to  conclude  that 


dE[MoJt)\3^r] 


dt 


=  e 


-pt 


dE[V{c,0^{t),^0^{t))\^r\ 


dt 


-  pE[V{cio^{t),^0^{t))  \  7r\ 


=  -  pV^(q(!).(t),  xo^(t))  |  3^r],  Vf  f  [r,0),  Vr  ^  0. 


where  ^  0}  is  the  natural  filtration  of  the  Markov  process  zq^  •=  other 

hand,  from  (33)  and  (37),  we  conclude  that 


Lorr.V{q,x)^pV{q,x),  Vg  G  Q,  x  G  M" 


and  therefore 


which  leads  to 


dE[Mo^{t)\3^r] 

dt 


^0, 


Mt  ^  [r,  oo),  Vr  ^  0, 


^  wpo  op 

E  [Mo^it]  I  ^r]  ^  MoM,  Vt  6  [r,  ®),  Vr  ^  0, 


and,  in  particular  for  r  =  0, 

r  n  wpo 

E  [Mo^(t)]  ^  Mo^(O)  =  V{qo,xo),  Vt  ^  0,  (50) 

confirming  that  (49)  is  indeed  a  non-negative  supermartingale.  It  then  follows  from  Doob’s  mar¬ 
tingale  inequality  that 

E  (3te[0,co)  :  e~P*V{c[o,ni't)^^o,ni't))  >  fj)  ^  >  0.  (51) 

To  establish  global  existence  of  solution,  we  use  the  fact  that  we  cannot  have  T^ax  <  oo  if  x(t) 
remains  in  a  bounded  set.  From  this  and  the  fact  that  the  Lyapunov  function  is  radially  unbounded 
(per  A14),  we  conclude  that  for  every  sample  path  for  which  T^ax  <  oo,  the  Lyapunov  function 
must  take  arbitrarily  large  values  in  the  (bounded)  interval  [0,Tniax),  i-G., 

Tmax  <00  ^  Vm  >  0,  6  [0,Tmax),E(qe',„(^),Xe)^(t))  ^  €^'^”'^^171 


and  therefore 

P(Tmax  <  oo)  ^  P  >  0,  e  [0,Tmax),E(qo,nW>xci^(t))  ^ 
^  P  (ym  >0,3te  [0,Tmax),E(qc»,nW)Xc)^(t))  ^ 
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^  P  {^m  >  0,3t  e  [0,oo),P(qo^(t),xci^(t))  ^ 

=  ^(n  [0,co),V  {ciOm{t),^Om{t))  > 

m>0 

where  we  used  the  fact  that  for  any  infinite  sequence  of  measurable  sets  Ei  E2  ^  ^  ^  we 

have  that  P(n^^i£’^)  =  lim^^oo  P(^n)-  If  we  now  use  (51)  with  /i  =  m,  we  conclude  that 

P(Tn,ax  <  00)  ^  lim  =0,  Vr  e  [0,  oo), 

m^oo  m 

which  means  that  the  solution  to  the  SDSHS  must  be  globally  defined  with  probability  one,  com¬ 
pleting  the  proof  of  P6. 

The  proof  of  P7  follows  from  (51)  with  p  =  0,  since  the  definitions  of  the  set  Om  and  the  stopped 
processes  allow  us  to  conclude  that,  for  every  /i  ^  m, 

3t  e  [0,co)  :V{ci{t),x{t))  ^  IJ,  e  [0,  go)  :  P(qo^(f),  xo^(f))  ^ 

and  therefore,  we  can  use  (51)  with  p  =  0  to  conclude  that 

P  (3t  e  [0,0))  :  P(q(t),x(t))  >^?j^F(3te  [0,o)  :  P(qo„,  W,  xo,„(t))  ^  /x)  T 
which  establishes  P7. 

The  proposition  P8  is  a  direct  consequence  of  (39),  which  can  be  used  to  provide  an  explicit 
estimate  on  the  state  bound  5,  as  a  function  of  p,  6  in  the  definition  in  (14):  for  given  6,p  >  0,  we 
simply  need  to  pick  p,  5  >  0  sufficiently  small  so  that 

||x(t)||  >  p  ^  y(q(t),x(t))  ^  p  and  ||xo||  ^  xq)  ^  ^ 

11 

The  existence  of  such  p,  5  >  0  is  guaranteed  by  (40),  as  we  can  pick  p  :=  7i(p)  and  5  <  7^^(ep). 
In  this  case,  for  every  initial  condition  q(0)  =  e  x(0)  =  xq  elSE  with  ||xo||  ^  5,  we  have  that 
(14)  holds. 

We  refer  the  reader  to  [19,  Theorem  2,p.  39]  for  the  proof  of  P9,  which  is  somewhat  more 
technical. 

5.4.2  Proof  of  Theorem  5 

The  bounds  on  the  expected  values  found  in  Theorem  5  are  obtained  by  first  using  Dynkin’s 
inequality  for  stopped  process  associated  with  sets  Om  of  the  form  (36)  and  then  making  m  ^  oo. 
Specifically,  to  establish  PIO  in  Theorem  5,  we  use  (50)  with  p  =  0  to  conclude  that 

^[V{^O^it),^0^{t))]  ^  Viqo,Xo),  Vt  ^  0. 

The  result  then  follows  from  proposition  P5  in  Lemma  1  [with  Vm  ■=  V,  Vm  and  c  :=  P(go,3:o)]. 
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To  prove  proposition  Pll  in  Theorem  5,  first  note  that 

LViq,x)^-W{q,x)  ^  Lo^V{q,x)^-WoJq,x):= 

We  therefore  conclude  from  (31)  in  Theorem  3  that,  \/t  ^  r  :=  0, 

^[v{q.0,n{t),^0At))  =  y{%,XQ)  +  ^^¥.^LOrnV{(\Orn{s),^Orn{s)) 

V{qo,XQ)  -  E  (qo,n(s)>xo^(s)) 


-W{q,x)  {q,x)eOm 
0  {q,x)^Om. 


ds 

ds 


and,  consequently. 


E  [Wo^(qo^(s),xc.^(s))  ds^V{qo,Xo),  Vt  ^  0. 


We  now  use  proposition  P5  in  Lemma  1  [with  Vm  •=  Wom  c  :=  l^(^05^o)]  to  conclude  that 


Te  [TE(q(s),x(s))lds  ^  Viqo,xo),  Vt  ^  0, 
Jo  L  -I 


from  which  (42)  follows. 

To  prove  proposition  P12  of  Theorem  5,  we  can  use  an  auxiliary  SDSHS  obtained  by  augmenting 
the  continuous  state  of  our  original  SDSHS  with  a  time  variable  T  that  is  initialize  at  T(0)  =  0, 
that  flows  according  to  T  =  1  and  that  is  not  affected  by  any  of  the  resets,  which  means  that 
T(t)  t,  Vt.  The  extended  generator  L  of  this  auxiliary  system  can  be  constructed  from  the 
extended  generator  L  of  the  original  system  using 

dV{q,x,T) 


LV{q,x,T)  LH(g,x,r)  + 


dT 


Vge  Q,xGM^,r^0, 


where  in  applying  the  extended  generator  L  of  the  original  system  to  H(g,x,T),  we  regard  T  as  a 
constant  parameter.  If  we  now  use  the  original  candidate  Lyapunov  function  V{q^x)  to  construct 
a  candidate  Lyapunov  function  V{q^x^T)  for  the  auxiliary  SDSHS  as  follows 

V{q,  x,  T)  :=  eP'^V{q,  x),  Mq  e  Q,x  ,T  e  [0,  oo), 

we  conclude  from  (43)  that 

LV{q,x,T)  ■■=  eP^{LV{q,x)  +  pV{q,x))  ^  e^^c,  Vg  e  Q,  a;  e  M”,  T  e  [0,oo). 

Using  (31)  for  the  stopped  processes  of  the  auxiliar  system  associated  with  sets  Om  of  the  form 
(36)  defined  using  the  new  Lyapunov  function  U(g,x,T),  we  conclude  that 


E 


[E(qci^(t),x(t)ei^,T(t))  ^  E(go,a;o,0)  +  eP^cds  =  V{qo,xo,0)  +  -  1),  Vt  ^  0. 


We  can  then  use  proposition  P5  in  Lemma  1  [with  Vm  •=  U,  Vm  and  c  :=  V (go?  ^o?  0)  +  cj p[e^^  —  1)] 
to  go  back  to  the  non-stopped  processes  and  conclude  that 

E[E(q(0,x(t),T(0)]  ^E(go,xo,0)  +  ^(e^'-l)  Vt  ^  0, 
from  which  (44)  follows. 

We  refer  the  reader  to  [19,  Theorem  6,  p.  50]  for  the  proof  of  P13,  which  is  a  variation  of  the 
proofs  discussed  above. 
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6  Conclusions  and  Open  Research  Directions 


Stochastic  hybrid  systems  provide  an  attractive  tool  to  model  and  analyze  networked  control  sys¬ 
tems  in  which  network-induced  events  (such  as  the  departure  and  arrival  of  network  packets) 
introduce  stochastic  instantaneous  changes  in  the  systems’  states.  We  presented  several  tools  for 
the  analysis  of  such  systems  and  illustrated  their  use  in  the  context  of  a  few  simple  examples. 

While  significant  progress  has  been  made  towards  the  analysis  of  SHSs,  several  important  prob¬ 
lems  remain  open:  The  stability  results  for  linear  TTSHS  are  fairly  complete,  but  the  study  of 
TTSHS  with  nonlinear  vector  fields  or  reset  maps  remains  mostly  open.  The  analysis  of  SDSHS 
also  presents  several  open  problems;  most  notably  the  development  of  systematic  methods  for  the 
construction  of  Lyapunov  functions,  perhaps  based  on  stochastic  counter  parts  of  backstepping  or 
similar  procedures.  A  notable  omission  in  this  paper  is  an  explicit  discussion  on  recurrence,  i.e., 
the  property  that  the  state  of  a  stochastic  system  returns  infinitely  often  to  a  particular  subset  of 
the  state  space  with  high  probability.  This  notion  has  proved  to  be  useful  in  numerous  contexts, 
including  the  introduction  of  stochastic  notions  of  input-to-state  stability  (ISS)  [23,  24]. 
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